[BUG] repository-s3 installation - Connection timed out

[toby181]

Describe the bug

The installation of the repository-s3 plugin fails with

./bin/opensearch-plugin install repository-s3
-> Installing repository-s3
-> Failed installing repository-s3
-> Rolling back repository-s3
-> Rolled back repository-s3
Exception in thread "main" java.net.ConnectException: Connection timed out
    at java.base/sun.nio.ch.Net.connect0(Native Method)
    at java.base/sun.nio.ch.Net.connect(Net.java:579)
    at java.base/sun.nio.ch.Net.connect(Net.java:568)
    at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
    at java.base/java.net.Socket.connect(Socket.java:633)
    at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
    at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:174)
    at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:183)
    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531)
    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636)
    at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
    at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:380)
    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)
    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179)
    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:142)
    at org.opensearch.plugins.InstallPluginCommand.urlExists(InstallPluginCommand.java:425)
    at org.opensearch.plugins.InstallPluginCommand.getOpenSearchUrl(InstallPluginCommand.java:392)
    at org.opensearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:309)
    at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:272)
    at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:249)
    at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
    at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
    at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
    at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
    at org.opensearch.cli.Command.main(Command.java:101)
    at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

I'm trying to install the repository-s3 plugin via different approaches.

1. Installation via Helm: https://github.com/opensearch-project/helm-charts/blob/opensearch-2.12.0/charts/opensearch/values.yaml#L473

values:
    extraEnvs:
    - name: HTTP_PROXY
      value: http://...
    - name: HTTPS_PROXY
      value: http://...
    - name: NO_PROXY
      value: "..."
    - name: http_proxy
      value: http://...
    - name: https_proxy
      value: http://...
    - name: no_proxy
      value: "..."
...
    ## Enable to add 3rd Party / Custom plugins not offered in the default OpenSearch image.
    plugins:
      enabled: true
      installList:
        - repository-s3

2. Manually connected to one of the OpenSearch pods and manually ran:

$ env | grep -i proxy
NO_PROXY=...
http_proxy=http://...
https_proxy=http://...
HTTPS_PROXY=http://...
no_proxy=....
HTTP_PROXY=http://...
$ ./bin/opensearch-plugin install repository-s3

OpenSearch version 2.7 running on kubernetes, installed via Helm.

Host/Environment (please complete the following information):

• OS: kubernetes
• Version 1.24

[prudhvigodithi]

Hey @toby181
The opensearch-plugin cli default pulls the plugin from https://artifacts.opensearch.org
example for repository-s3:2.8.0 the URL is https://artifacts.opensearch.org/releases/plugins/repository-s3/2.8.0/repository-s3-2.8.0.zip, can you make you sure if your exposed proxy has this URL whitelisted ?

Also there are multiple was to install a plugin
sudo ./opensearch-plugin install https://<PRIVATE_URL>/repository-s3-2.8.0.zip this way you can download and store the zip in your private proxy repo and use that during plugin installation .

@peterzhuamazon @bbarani

[toby181]

Hey @prudhvigodithi
Thanks for your feedback and the URL.

I did some more tests using curl in combination with our proxy within the pod.

proxy configuration inside pod

$ env | grep -i proxy
NO_PROXY=...
HTTPS_PROXY=http://...
HTTP_PROXY=http://...

Successful download using curl within the pod

$ curl https://artifacts.opensearch.org/releases/plugins/repository-s3/2.7.0/repository-s3-2.7.0.zip --output repository-s3-2.7.0.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 5356k  100 5356k    0     0  17.7M      0 --:--:-- --:--:-- --:--:-- 17.7M
$ ls -l
total 5360
-rw-rw-r-- 1 opensearch opensearch 5485325 Jun 12 13:50 repository-s3-2.7.0.zip

Failed installation by using opensearch-plugin cli with the given URL

$ ./bin/opensearch-plugin install https://artifacts.opensearch.org/releases/plugins/repository-s3/2.7.0/repository-s3-2.7.0.zip
-> Installing https://artifacts.opensearch.org/releases/plugins/repository-s3/2.7.0/repository-s3-2.7.0.zip
-> Downloading https://artifacts.opensearch.org/releases/plugins/repository-s3/2.7.0/repository-s3-2.7.0.zip
-> Failed installing https://artifacts.opensearch.org/releases/plugins/repository-s3/2.7.0/repository-s3-2.7.0.zip
-> Rolling back https://artifacts.opensearch.org/releases/plugins/repository-s3/2.7.0/repository-s3-2.7.0.zip
-> Rolled back https://artifacts.opensearch.org/releases/plugins/repository-s3/2.7.0/repository-s3-2.7.0.zip
Exception in thread "main" java.net.ConnectException: Connection timed out
	at java.base/sun.nio.ch.Net.connect0(Native Method)
	at java.base/sun.nio.ch.Net.connect(Net.java:579)
	at java.base/sun.nio.ch.Net.connect(Net.java:568)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:633)
	at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
	at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:174)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:183)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636)
	at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
	at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:380)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
	at org.opensearch.plugins.InstallPluginCommand.downloadZip(InstallPluginCommand.java:455)
	at org.opensearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:340)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:272)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:249)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

I did not yet test the installation when we store the zip in an internal repo but this would required a pre-step that does the down- and upload everytime a new version is available and we have to take care that the version is updated in the HelmRelease as well.

[prudhvigodithi]

Hey @toby181 can you try the following ways and update back which one worked?

1. ./bin/opensearch-plugin install -DproxyHost=<your_proxy_host> -DproxyPort=<your_proxy_port> repository-s3
2. During the cluster startup set the OPENSEARCH_JAVA_OPTS to -Dhttp.proxyHost=<your_proxy_host> -Dhttp.proxyPort=<your_proxy_port> -Dhttps.proxyHost=<your_proxy_host> -Dhttps.proxyPort=<your_proxy_port>
3. Since you are able to download the file
   ./bin/opensearch-plugin install file:/<path_to_zip_file_downloaded>
   Thank you

[toby181]

Hey @prudhvigodithi

Here we go...

1. Regular proxy environment variables are disabled, proxy variable are added to opensearchJavaOpts and the installation of the plugin is done within the HelmRelease:

values:
    extraEnvs:
    #- name: HTTP_PROXY
    #  value: http://...
    #- name: HTTPS_PROXY
    #  value: http://...
    #- name: NO_PROXY
    #  value: "..."
...
    opensearchJavaOpts: "-Xms6g -Xmx6g -Dhttp.proxyHost=proxy.example.com -Dhttp.proxyPort=3128 -Dhttps.proxyHost=proxy.example.com -Dhttps.proxyPort=3128 -Dhttp.nonProxyHosts=.example.com,..."
...
    ## Enable to add 3rd Party / Custom plugins not offered in the default OpenSearch image.
    plugins:
      enabled: true
      installList:
        - repository-s3

Checking the log file of the pod:

-> Installing repository-s3
-> Downloading repository-s3 from opensearch
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission setDefaultAuthenticator
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission opensearch.allow_insecure_settings read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed repository-s3 with folder name repository-s3
Disabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
Enabling OpenSearch Security Plugin

Seems like a success, although I'm concerned about the warning. As I do not yet whether this will affect my setup, I can't tell whether this is good or bad.

2. No proxy environment variables nor proxy settings via opensearchJavaOpts. The installation is done manually within the pod using the opensearch-plugin cli.

values:
    extraEnvs:
    #- name: HTTP_PROXY
    #  value: http://...
    #- name: HTTPS_PROXY
    #  value: http://...
    #- name: NO_PROXY
    #  value: "..."
...
    opensearchJavaOpts: "-Xms6g -Xmx6g"
...
    ## Enable to add 3rd Party / Custom plugins not offered in the default OpenSearch image.
    plugins:
      enabled: false
      installList:
        - repository-s3

pod:

(wrong syntax)
$ ./bin/opensearch-plugin install  -DproxyHost=proxy.example.com -DproxyPort=3128 repository-s3
A tool for managing installed opensearch plugins

Non-option arguments:
[String] -- command

Option             Description
------             -----------
-E <KeyValuePair>  Configure a setting
-h, --help         Show help
-s, --silent       Show minimal output
-v, --verbose      Show verbose output
ERROR: Unknown command [install??]


(correct syntax - failing - using -DproxyHost)
$ ./bin/opensearch-plugin -E -DproxyHost=proxy.example.com -E -DproxyPort=3128 install repository-s3
-> Installing repository-s3
-> Failed installing repository-s3
-> Rolling back repository-s3
-> Rolled back repository-s3
Exception in thread "main" java.net.ConnectException: Connection timed out


(correct syntax - failing - using -Dhttps.proxyHost)
$ ./bin/opensearch-plugin -E -Dhttps.proxyHost=proxy.example.com -E -Dhttps.proxyPort=3128 install repository-s3
-> Installing repository-s3
-> Failed installing repository-s3
-> Rolling back repository-s3
-> Rolled back repository-s3
Exception in thread "main" java.net.ConnectException: Connection timed out


(correct syntax - failing - using -Dhttps.proxyHost and -Dhttps.proxyHost)
$ ./bin/opensearch-plugin -E -Dhttp.proxyHost=proxy.example.com -E -Dhttp.proxyPort=3128 -E -Dhttps.proxyHost=proxy.example.com -E -Dhttps.proxyPort=3128 install repository-s3
-> Installing repository-s3
-> Failed installing repository-s3
-> Rolling back repository-s3
-> Rolled back repository-s3
Exception in thread "main" java.net.ConnectException: Connection timed out

Using the opensearch-plugin cli I'm getting a timeout with different configurations for the proxy, so I guess my parameters are not correct?!

3. Installation via internal repository
   Download the zip file beforehand and upload to our internal repository.

$ ./bin/opensearch-plugin install https://internalrepo.example.com/repos/opensearch-artifacts/raw/repository-s3/repository-s3-2.7.0.zip?at=refs%2Fheads%2Fmaster
-> Installing https://internalrepo.example.com/repos/opensearch-artifacts/raw/repository-s3/repository-s3-2.7.0.zip?at=refs%2Fheads%2Fmaster
-> Downloading https://internalrepo.example.com/opensearch-artifacts/raw/repository-s3/repository-s3-2.7.0.zip?at=refs/heads/master
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission setDefaultAuthenticator
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission opensearch.allow_insecure_settings read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]
...

I did not continue with the installation since I would like to install it via Helm but I'd assume that it succeeds.

[prudhvigodithi]

Thanks @toby181
The best solution is to use lifecycle.postStart and add the install command.

You can use batch mode to install a plugin without a prompt

 ./bin/opensearch-plugin install  --batch https://internalrepo.example.com/repos/opensearch-artifacts/raw/repository-s3/repository-s3-2.7.0.zip?at=refs%2Fheads%2Fmaster

Thank you

[toby181]

Thanks @prudhvigodithi !
What's the difference to this?

    ## Enable to add 3rd Party / Custom plugins not offered in the default OpenSearch image.
    plugins:
      enabled: true
      installList:
        - https://internalrepo.example.com/repos/opensearch-artifacts/raw/repository-s3/repository-s3-2.7.0.zip?at=refs%2Fheads%2Fmaster

I did not test that with the s3 plugin but I did in the past with the prometheus-exporter.

Any idea about the correct syntax for the opensearch-plugin cli in terms of environment variables? My idea is to call the opensearch-plugin cli in the postStart.