AttributeError: module '__main__' has no attribute '_module_fqn'

[anxstj]

• Which version of Ansible are you running?

ansible [core 2.11.2] 
  config file = ~/.ansible.cfg
  configured module search path = ['~/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/venv/lib/python3.9/site-packages/ansible
  ansible collection location = ~/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/venv/bin/ansible
  python version = 3.9.6 (default, Jul 16 2021, 00:00:00) [GCC 11.1.1 20210531 (Red Hat 11.1.1-3)]
  jinja version = 3.0.1
  libyaml = True

• Is your version of Ansible patched in any way? no
• Are you running with any custom modules, or module_utils loaded? no
• Have you tried the latest master version from Git? yes
• Mention your host and target OS and versions
  • host: Fedora 34
  • target: Debian 10 docker image
• Mention your host and target Python versions
  • host: Python 3.9
  • target: Python 3.7

Example Playbook:

---
- name: Converge
  hosts: all

  tasks:
    - name: install vim
      ansible.builtin.apt:
        name: vim
        state: present

Error:

The full traceback is:
Traceback (most recent call last):
  File "master:/usr/local/share/mitogen/ansible_mitogen/runner.py", line 975, in _run
    self._run_code(code, mod)
  File "master:/usr/local/share/mitogen/ansible_mitogen/runner.py", line 939, in _run_code
    exec(code, vars(mod))
  File "master:/usr/local/venv/lib/python3.9/site-packages/ansible/modules/apt.py", line 1310, in <module>
  File "master:/usr/local/venv/lib/python3.9/site-packages/ansible/modules/apt.py", line 1114, in main
  File "master:/usr/local/venv/lib/python3.9/site-packages/ansible/module_utils/common/respawn.py", line 39, in respawn_module
    payload = _create_payload()
  File "master:/usr/local/venv/lib/python3.9/site-packages/ansible/module_utils/common/respawn.py", line 76, in _create_payload
    module_fqn = sys.modules['__main__']._module_fqn
AttributeError: module '__main__' has no attribute '_module_fqn'
fatal: [linux-debian-10]: FAILED! => {
    "ansible_facts": {},
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"master:/usr/local/share/mitogen/ansible_mitogen/runner.py\", line 975, in _run\n    self._run_code(code, mod)\n  File \"master:/usr/local/share/mitogen/ansible_mitogen/runner.py\", line 939, in _run_code\n    exec(code, vars(mod))\n  File \"master:/usr/local/venv/lib/python3.9/site-packages/ansible/modules/apt.py\", line 1310, in <module>\n  File \"master:/usr/local/venv/lib/python3.9/site-packages/ansible/modules/apt.py\", line 1114, in main\n  File \"master:/usr/local/venv/lib/python3.9/site-packages/ansible/module_utils/common/respawn.py\", line 39, in respawn_module\n    payload = _create_payload()\n  File \"master:/usr/local/venv/lib/python3.9/site-packages/ansible/module_utils/common/respawn.py\", line 76, in _create_payload\n    module_fqn = sys.modules['__main__']._module_fqn\nAttributeError: module '__main__' has no attribute '_module_fqn'\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

Verbose output:
verbose.log.txt

Workarounds like:

1. Adding vars: { mitogen_task_isolation: fork } to the task
2. Adding ansible.builtin.apt to ALWAYS_FORK_MODULES in ansible_mitogen/planner.py

doesn't work.

[baszoetekouw]

I'm trying to look into this. The issue seems to be caused by the introduction of the module_respawn API in ansible in commit 4c5ce5a1a9e79a845aff4978cfeb72a0d4ecf7d6, an in particular by the change on line 197 of module_common.py which changes the invocation of ansible's run_module() from

runpy.run_module(mod_name='%(module_fqn)s', init_globals=None, run_name='__main__', alter_sys=True)

to

runpy.run_module(mod_name='%(module_fqn)s', init_globals=dict(_module_fqn='%(module_fqn)s', _modlib_path=modlib_path),
                         run_name='__main__', alter_sys=True)

In particular, it seems that mitogen_ansible doesn't insert _module_fqn into the target module's globals.

My guess is that that attribute should be inserted into the target module somewhere around here https://github.com/mitogen-hq/mitogen/blob/master/ansible_mitogen/runner.py#L970 but simply adding

mod._module_fqn = self.py_module_name

there has no effect.

I'm lacking knowledge of python internals in order to get mitogen to insert this _module_fqn attribute (and _modlb_path, which is used in the next line) into the __main__ module of the executed script.

[Links2004]

can confirm the lastest master still has this problem

[philfry]

Happens on a Fedora 35 controller node w/ OracleLinux 8.5 managed nodes using the dnf module as well. The funny thing is: it happens randomly, not everytime.

[phemmer]

My guess is that that attribute should be inserted into the target module somewhere around here https://github.com/mitogen-hq/mitogen/blob/master/ansible_mitogen/runner.py#L970 but simply adding

mod._module_fqn = self.py_module_name

there has no effect.

I put it here. Seems to have addressed the issue.

mitogen/ansible_mitogen/runner.py

Lines 526 to 528 in a564d8a

	mod = sys.modules.setdefault(fullname, imp.new_module(fullname))
	mod.__file__ = "master:%s" % (path,)
	mod.__loader__ = self

[baszoetekouw]

@phemmer What exactly have you inserted there? This code doesn't even seem to be called when running a module such as apt (at least, putting a generic exception there had no effect whatsoever).

[mhenniges]

I think this issue is triggered when the ansible_python_interpreter being used on the target is not the system python. The ansible modules that deal with packages require python libraries that are not installable with pip, and can't reasonable be streamed out by mitogen because of that. In the ansible module code, there's a breakout were it will respawn into the system python in order to find these libraries: https://github.com/ansible/ansible/blob/6e57c8c0844c44a102dc081b93d5299cae9619e2/lib/ansible/module_utils/common/respawn.py#L18
And I suspect something about that causes the problem.
For anybody else having this issue, if you 1) make sure ansible is using the system python on the far end, and 2) perhaps - make sure that you don't have the python module locally so that mitogen doesnt try to send it; does the issue go away?

[baszoetekouw]

@mhenniges I think the situation you describe is correct, but it still should be supported by mitogen, I think. I don't think it is hard to solve (you need to insert two globals in the target module's main namespace), I only don't know enough about the mitogen internals to determine how/where to do that.

[anxstj]

@mhenniges Explicitly setting ansible_python_interpreter=/usr/bin/python (or auto_legacy) for the Debian 10 container circumvents the problem.

Setting it to /usr/bin/python3 (or auto) results in the same error.

I'm testing this with molecule and it sets interpreter_python = auto_silent per default, so that's probably the reason why it is breaking in my case.

[philfry]

I have INTERPRETER_PYTHON set to auto on the OEL 8.x managed node. Even though there is only one python version installed, there are lots of symlinks that are needed to be followed for the final binary:

/usr/bin/python3 → 
/etc/alternatives/python3 →
/usr/bin/python3.6 →
/usr/libexec/platform-python3.6

Ansible uses /usr/libexec/platform-python though, which is a symlink to /usr/libexec/platform-python3.6, because of the distro map:

INTERPRETER_PYTHON_DISTRO_MAP(default) = {
  'centos': {
    '6': '/usr/bin/python',
    '8': '/usr/libexec/platform-python',
    '9': '/usr/bin/python3'
  },
  'debian': {
    '8': '/usr/bin/python',
    '10': '/usr/bin/python3'
  },
  'fedora': {
    '23': '/usr/bin/python3'
  },
  'oracle': {
    '6': '/usr/bin/python',
    '8': '/usr/libexec/platform-python',
    '9': '/usr/bin/python3'
  },
  'redhat': {
    '6': '/usr/bin/python',
    '8': '/usr/libexec/platform-python',
    '9': '/usr/bin/python3'
  },
  'rhel': {
    '6': '/usr/bin/python',
    '8': '/usr/libexec/platform-python',
    '9': '/usr/bin/python3'
  },
  'ubuntu': {
    '14': '/usr/bin/python',
    '16': '/usr/bin/python3'
  }
}

As already mentioned, for me, the error only occurs in the dnf module. And only from time to time.

[baszoetekouw]

I think you are missing the point of this bug; sure, I'd be nice to have a workaround by setting the python interpreter to a different binary, but that still wouldn't fix this bug.

The problem is that the ansible API has changed, and this breaks specific mitogen workflows. The solution should thus be to fix mitogen, not to avoid the specific circumstances in which this workflow is triggered.

[baszoetekouw]

I'm not sure what has happened, but eveything now Just Work with ansible-core 2.12.2 and mitogen 0.3.2.

I'm not sure what has happened, but eveything now Just Work with ansible-core 2.12.2 and mitogen 0.3.2.

Worked for few runs, and then started to spit out the same error. Also, some other similar modules, like debconf or dnf fails.

[rfc1036]

I have experienced the same bug when connecting from a Debian unstable system (Debian mitogen 0.3.1-3) to a Debian 11 system. I fixed it by installing on the target these packages (I did not check which one was actually relevant): python3-chardet python3-decorator python3-pkg-resources python3-requests.