Add a robots.txt file to public ressources with SSO enabled

[Root-Core]

Summary

Serving a robots.txt file can prevent the indexing of publicly available but not openly accessible resources.
If SSO is enabled, the Pangolin authentication page is served and indexed by crawlers.
This increases the security threat surface and is likely unwanted behaviour.

Motivation

As @miloschwartz wrote: We are also aware that the search engines are crawling the pangolin auth pages

If a resource is protected with SSO, it's unlikely that it's meant to be indexed by search engines.

Proposed Solution

Serve a robots.txt that instructs the crawler to skip indexing this website / resource.

Alternatives Considered

Add meta tags and/or HTTP headers.
However, this has the disadvantage that the page is requested by the crawler and that it only applies to a single page.

Additional Context

No response

[LaurenceJJones]

We would need this to be customized in some manner, as some people are using different routes to be protected by SSO, EG: allow most routes but /admin is SSO'ed. We cant just presume to hijack the robots.txt in all cases.

[Root-Core]

In that case the routes could be added to the robots.txt, which would expose them or add the X-Robots-Tag header.

Adding the path to the robots.txt wouldn't really add attack surface, as the path is already known when it's crawled.
So this concern is only valid if the path is not linked, which isn't unlikely.

I would argue in favor of the header for rules and robots.txt if the ressource uses SSO by default.
That's the best of both worlds.