Skip to content

filter_tasks_by_metadata masks connection/timeout errors with AttributeError (catches bare Exception, reads .http_code) #3316

Description

@prabhaharanv

Description

ServiceMetadataProvider.filter_tasks_by_metadata
(metaflow/plugins/metadata_providers/service.py) catches bare Exception and
then reads e.http_code:

try:
    resp, _ = cls._request(None, url, "GET")
except Exception as e:
    if e.http_code == 404:
        ...
    raise e

http_code is only defined on ServiceException — every other handler in the
file uses except ServiceException as ex. When cls._request() raises a
non-ServiceException (e.g. a requests ConnectionError/Timeout, which
_request re-raises unwrapped on its final retry via the bare
except: ... raise at the end of its retry loop), reading e.http_code raises
AttributeError, replacing the real network error with a confusing one.

Steps to reproduce

Point METAFLOW_SERVICE_URL at an unreachable host and trigger a Client
operation that routes through filter_tasks_by_metadata. Instead of the
underlying connection error, the call fails with
AttributeError: 'ConnectionError' object has no attribute 'http_code'.

Minimal mechanism (no Metaflow import needed):

class ServiceException(Exception):
    def __init__(self, msg, http_code=None):
        self.http_code = http_code

def handler(exc):                     # current behavior
    try:
        raise exc
    except Exception as e:            # <-- too broad
        if e.http_code == 404:        # <-- AttributeError on non-ServiceException
            return "internal-error"
        raise

handler(ConnectionError("refused"))   # -> AttributeError, not ConnectionError

Expected vs. actual

  • Expected: the underlying ConnectionError/Timeout propagates so the user
    sees the real cause.
  • Actual: an AttributeError masks it.

Root cause

except Exception as e catches everything; e.http_code assumes a
ServiceException. _request only attaches http_code when it constructs a
ServiceException from an HTTP response — connection/timeout errors from
requests propagate unwrapped, so they reach this handler without an
http_code attribute.

Proposed fix

Narrow the catch to except ServiceException as e (matching every other handler
in the file), so non-service errors propagate unchanged. A PR follows with a
regression test.

Environment

  • Metaflow version: 2.19.35 (also present on current master)
  • File: metaflow/plugins/metadata_providers/service.py

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions