Univention Bugzilla – Bug 55632
fix leftover from Python 3 migration
Last modified: 2023-06-11 10:52:57 CEST
Some leftover references of Python 2.7 exists: management/univention-appcenter/conffiles/etc/univention/service.info/services/univention-appcenter.cfg: > programs=python /usr/bin/univention-run-app %s start mail/univention-spamassassin/debian/univention-spamassassin.postinst: > python -c 'import pty; pty.spawn(["/etc/cron.daily/spamassassin"]) management/univention-appcenter/debian/univention-management-console-module-appcenter.postinst: > python -c "from univention.appcenter.actions import get_action; get_action… management/univention-management-console/usr/share/univention-management-console/saml/update_metadata: > metadata=$(python -c "from saml2.metadata import create_metadata_string…
See https://git.knut.univention.de/groups/univention/-/epics/10 and https://luns.knut.univention.de/etherpad/p/python3-migration-todo
base/univention-lib/debian/python3-univention-lib.postinst: python <<EOL
base/univention-updater/script/check.sh: fnregex="$(python -c 'import re,sys;print re.escape(sys.argv[1])' "$1")" doc/developer-reference/appendix/debian-packaging.rst:``${python:Depends}`` mail/univention-mail-postfix/share/listfilter.py: # read from stdin python -u is required for unbuffered streams management/univention-appcenter/debian/univention-management-console-module-apps.preinst: echo 'from univention.management.console.modules.appcenter.app_center import Application; Application.sync_with_server()' | python packaging/univention-l10n/univention-ucs-translation-build-package:#!/usr/bin/env python packaging/univention-l10n/univention-ucs-translation-fakemessage:#!/usr/bin/env python packaging/univention-l10n/univention-ucs-translation-merge:#!/usr/bin/env python packaging/univention-l10n/univention/l10n/__init__.py:#!/usr/bin/env python packaging/univention-l10n/univention/l10n/helper.py:#!/usr/bin/env python packaging/univention-l10n/univention/l10n/l10n.py:#!/usr/bin/env python packaging/univention-l10n/univention/l10n/message_catalogs.py:#!/usr/bin/env python packaging/univention-l10n/univention/l10n/sourcefileprocessing.py:#!/usr/bin/env python test/product-tests/base/utils.sh: python <<%EOF test/ucs-test/tests/00_checks/01_univention_system_check:python ./univention-system-check test/utils/utils.sh: python -m shared-utils/license_client "${lb}" -u "$users" "$(date -d '+6 month' '+%d.%m.%Y')" test/utils/utils.sh: python -m shared-utils/license_client "${lb}" -u "$users" "$(date -d '+6 month' '+%d.%m.%Y')" test/utils/utils.sh: python -m shared-utils/apps "$@" test/utils/utils.sh: latest="$(python -m shared-utils/app-info -a "$app" -v)"
base/univention-system-setup/umc/python/setup/checks/repositories.py: # FIXME: When changing to Python 3 use process.args here management/univention-directory-manager-rest/src/univention/admin/rest/module.py: tornado.httputil.responses[422] = 'Unprocessable Entity' # Python 2 is missing this status code test/ucs-test/univention/testing/udm.py: from inspect import getargspec # Python 2
(In reply to Florian Best from comment #4) > test/utils/utils.sh: python -m shared-utils/license_client "${lb}" -u "$users" "$(date -d '+6 month' '+%d.%m.%Y')" > test/utils/utils.sh: python -m shared-utils/license_client "${lb}" -u "$users" "$(date -d '+6 month' '+%d.%m.%Y')" > test/utils/utils.sh: python -m shared-utils/apps "$@" > test/utils/utils.sh: latest="$(python -m shared-utils/app-info -a "$app" -v)" They called tools all come from univention/dist/ucs-ec2-tools> <https://git.knut.univention.de/univention/dist/ucs-ec2-tools/-/tree/master/shared-utils>. They are also called from UCS 4.4 test; they must remain compatible with that. Which Python interpreter to use should be an implementation detail; leave it to the hash-bang to select the right one and please do not hard-code it at the calling side. DIST/ucs-ec2-tools/shared-utils/license_client.py is already using `python3`. (In reply to Florian Best from comment #5) > base/univention-system-setup/umc/python/setup/checks/repositories.py: # FIXME: When changing to Python 3 use process.args here While "subprocess.Popen.args" is there, it is an undocumented detail and should not be used. See https://git.knut.univention.de/univention/ucs/-/merge_requests/719 (In reply to Florian Best from comment #3) > base/univention-lib/debian/python3-univention-lib.postinst: python <<EOL should be removed as it is old upgrade code.
*** Bug 54893 has been marked as a duplicate of this bug. ***
univention-updater.yaml 9e657c6f87cf | Bug #55632: migrated Python 2.7 leftovers and improved Python 3.11 support of univention-updater univention-updater (15.0.6-6) 9e657c6f87cf | Bug #55632: migrated Python 2.7 leftovers and improved Python 3.11 support of univention-updater univention-system-setup.yaml d34605ac9ea1 | Bug #55632: improved the Python 3.11 compatiblity of univention-system-setup univention-system-setup (13.0.6-3) d34605ac9ea1 | Bug #55632: improved the Python 3.11 compatiblity of univention-system-setup univention-squid.yaml 75d57ecb6042 | Bug #55632: improved the Python 3.11 compatiblity of univention-squid univention-squid (13.0.7-3) 75d57ecb6042 | Bug #55632: improved the Python 3.11 compatiblity of univention-squid univention-spamassassin (11.0.1-2) 2e5b50a6f50e | fix(SA): Use Python 3 to spawn rule update from postint univention-self-service (5.0.6-4) 6e9cd7afe9f4 | Bug #55632: improved the Python 3.11 compatiblity of univention-self-service univention-s4-connector.yaml 9e9b7fc4713b | Bug #55632: improves the way the s4 connector checks if it is running univention-s4-connector (14.0.13-6) 9e9b7fc4713b | Bug #55632: improves the way the s4 connector checks if it is running univention-radius (7.0.5-3) 724315863a20 | Bug #55632: Migrated distutils -> setuptools univention-python-heimdal (10.0.1-1) 724315863a20 | Bug #55632: Migrated distutils -> setuptools 0645c0aa62a6 | Bug #55632: remove unused X-Python-Version univention-python (13.0.3-1) 724315863a20 | Bug #55632: Migrated distutils -> setuptools univention-printserver.yaml ed09be1edf72 | Bug #55632: improved the Python 3.11 compatiblity of univention-printserver univention-printserver (13.0.8-3) ed09be1edf72 | Bug #55632: improved the Python 3.11 compatiblity of univention-printserver univention-portal.yaml 6362515184df | Bug #55632: improved the Python 2.7 compatiblity of univention-portal univention-portal (4.0.9-5) 6362515184df | Bug #55632: improved the Python 2.7 compatiblity of univention-portal univention-monitoring-client.yaml 1edd7f492513 | Bug #55632: improved the Python 3.11 compatiblity of univention-monitoring-client univention-monitoring-client (1.0.2-2) 1edd7f492513 | Bug #55632: improved the Python 3.11 compatiblity of univention-monitoring-client univention-management-console.yaml 08ff808f58d0 | Bug #55632: improved Python 3.11 compatiblity and metadata handling of univention-management-console univention-management-console-module-udm.yaml d3ec1dc3abb6 | Bug #55632: improved the Python 3.11 compatiblity of the UDM module for UMC univention-management-console-module-udm (10.0.5-3) d3ec1dc3abb6 | Bug #55632: improved the Python 3.11 compatiblity of the UDM module for UMC univention-management-console-module-diagnostic.yaml dfe14edc6c1a | Bug #55632: improved the Python 3.11 compatiblity of the UMC diagnostic module univention-management-console-module-diagnostic (6.0.5-3) dfe14edc6c1a | Bug #55632: improved the Python 3.11 compatiblity of the UMC diagnostic module univention-management-console (12.0.17-14) 08ff808f58d0 | Bug #55632: improved Python 3.11 compatiblity and metadata handling of univention-management-console univention-licence-python (11.0.1-1) 724315863a20 | Bug #55632: Migrated distutils -> setuptools 0645c0aa62a6 | Bug #55632: remove unused X-Python-Version univention-lib.yaml 6b9e26ad22e3 | Bug #55632: Python 3.11 compatiblity added to mime type handling of uniention-lib univention-lib (9.0.14-3) 6b9e26ad22e3 | Bug #55632: Python 3.11 compatiblity added to mime type handling of uniention-lib univention-lib (9.0.14-2) ede51d2ec46e | test(univention-lib): move import of test libraries into pytest fixture univention-l10n.yaml d76a6b69f8c3 | Bug #55632: updated hashbangs of univention-l10n univention-l10n (7.0.2-2) d76a6b69f8c3 | Bug #55632: updated hashbangs of univention-l10n univention-join.yaml 89305b83a2cd | Bug #55632: improved the Python 3.11 compatiblity of univention-join univention-join (12.0.5-4) 89305b83a2cd | Bug #55632: improved the Python 3.11 compatiblity of univention-join univention-ipcalc (11.0.1-1) 724315863a20 | Bug #55632: Migrated distutils -> setuptools univention-home-mounter.yaml 2d1088cf5ce9 | Bug #55632: improved the Python 3.11 compatiblity of univention-home-mounter univention-home-mounter (11.0.2-2) 2d1088cf5ce9 | Bug #55632: improved the Python 3.11 compatiblity of univention-home-mounter univention-grub.yaml 746652358a07 | Bug #55632: improved the Python 3.11 compatiblity of univention-grub univention-grub (13.0.2-2) 746652358a07 | Bug #55632: improved the Python 3.11 compatiblity of univention-grub univention-directory-manager-rest.yaml 94e4a56ee5ec | Bug #55632: Removed useless Python 2.7 compatiblity code from UDM REST API univention-directory-manager-rest (10.0.6-4) 94e4a56ee5ec | Bug #55632: Removed useless Python 2.7 compatiblity code from UDM REST API univention-directory-manager-modules.yaml c62d968c3585 | Bug #55632: improved the Python 3.11 compatiblity and the timezone handling univention-directory-manager-modules (15.0.20-7) c62d968c3585 | Bug #55632: improved the Python 3.11 compatiblity and the timezone handling univention-directory-listener.yaml a2d810b3c134 | Bug #55632: improved the Python 3.11 compatiblity of univention-directory-listener univention-directory-listener (14.0.8-3) a2d810b3c134 | Bug #55632: improved the Python 3.11 compatiblity of univention-directory-listener univention-debug-python (12.0.2-1) 724315863a20 | Bug #55632: Migrated distutils -> setuptools univention-debhelper (2.0.1-1) 724315863a20 | Bug #55632: Migrated distutils -> setuptools univention-config-registry.yaml 0e2d57267cb6 | Bug #55632: improved the Python 3.11 compatiblity of univention-config-registry univention-config-registry (15.0.10-2) 0e2d57267cb6 | Bug #55632: improved the Python 3.11 compatiblity of univention-config-registry univention-config-registry (15.0.10-1) cc5021ebe0b8 | Bug #55632: adjust pytests to new pytest version univention-appcenter.yaml 9da2a09a7986 | Bug #55632: migrated Python 2.7 leftovers and improved Python 3.11 support of univention-appcenter univention-appcenter (9.0.7-8) 9da2a09a7986 | Bug #55632: migrated Python 2.7 leftovers and improved Python 3.11 support of univention-appcenter univention-ad-connector.yaml dbe95b623994 | Bug #55632: improved the way the ad connector checks if it is running univention-ad-connector (14.0.13-7) dbe95b623994 | Bug #55632: improved the way the ad connector checks if it is running ucs-test (10.0.10-64) 1b7e605f8f6c | Bug #55632: improved the Python 3.11 compatiblity of test/ucs-test ucs-test (10.0.10-44) bd4bd4af73a0 | fixup! Bug #55632: migrated leftover Python 2 to Python 3 and removed unused import for Python 2.7 from test 10f454648dba | Bug #55632: migrated leftover Python 2 to Python 3 and removed unused import for Python 2.7 from test ucs-test (10.0.10-35) 890f6cc2681a | Bug #55632: document Python 3 hashbang python-keycloak (0.24.1-1) 724315863a20 | Bug #55632: Migrated distutils -> setuptools NONE 855b83d26d2f | Bug #55632: improved the Python 3.11 compatiblity of test/ucs-gui-tests 6465143dcbd7 | Bug #55632: utils.sh now does not anymore force an interpreter 4a8b3de73fba | Bug #55632: use Python 3 in product tests e004584699dc | Bug #55632: document substitution variable ${python3:Depends} over ${python:Depends} 3ef6611ed595 | Bug #55632: univention-appcenter-control uses Python 3.7 since a while
I see the following tracebacks in Jenkins: Traceback (most recent call last): File "/usr/share/univention-appcenter-listener-converter", line 203, in main find_and_convert_files(app) File "/usr/share/univention-appcenter-listener-converter", line 179, in find_and_convert_files if convert(app, dumped, filename, lo, pos): File "/usr/share/univention-appcenter-listener-converter", line 119, in convert objs = search_objects(udm_type, lo, pos, entryUUID=entry_uuid) File "/usr/lib/python3/dist-packages/univention/appcenter/udm.py", line 120, in search_objects module = _get_module(_module, _lo, _pos) File "/usr/lib/python3/dist-packages/univention/appcenter/udm.py", line 66, in _get_module _update_modules() File "/usr/lib/python3/dist-packages/univention/appcenter/udm.py", line 62, in _update_modules udm_modules.update() File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 153, in update _walk(root, w_root, w_files) File "/usr/lib/python3/dist-packages/univention/admin/modules.py", line 140, in _walk m = importlib.import_module('univention.admin.handlers.%s' % (modulepackage,)) # type: Any File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/univention/admin/handlers/users/passwd.py", line 38, in <module> import univention.admin.handlers.users.user File "/usr/lib/python3/dist-packages/univention/admin/handlers/users/user.py", line 76, in <module> utc = zoneinfo.ZoneInfo('UTC') AttributeError: module 'zoneinfo' has no attribute 'ZoneInfo' → strange. Should not happen with Python 3.7. What is causing it? Traceback (most recent call last): File "/usr/share/ucs-test/20_appcenter/100_settings.py", line 361, in test_file_setting_docker assert not os.path.exists(docker_file) File "/usr/lib/python3.7/genericpath.py", line 19, in exists os.stat(path) TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType Traceback (most recent call last): File "/usr/share/ucs-test/20_appcenter/100_settings.py", line 208, in test_string_setting_docker assert docker_shell(app, 'grep "test/setting: " /etc/univention/base.conf') == 'test/setting: My value\n' File "/usr/share/ucs-test/20_appcenter/100_settings.py", line 65, in docker_shell return subprocess.check_output(['docker', 'exec', container, '/bin/bash', '-c', command], stderr=subprocess.STDOUT, text=True) File "/usr/lib/python3.7/subprocess.py", line 395, in check_output **kwargs).stdout File "/usr/lib/python3.7/subprocess.py", line 472, in run with Popen(*popenargs, **kwargs) as process: File "/usr/lib/python3.7/subprocess.py", line 775, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.7/subprocess.py", line 1453, in _execute_child restore_signals, start_new_session, preexec_fn) TypeError: expected str, bytes or os.PathLike object, not NoneType → I hope they aren't influenced by Bug #56072. It could be that the VerifiedHTTPSConnection change is responsible for this.
The DVD Installation tests are failing due to d34605ac9ea18112b538782ae65d348d1e742ced. >>> from univention.management.console.modules.setup.netconf.modules import RunPhases >>> phases = RunPhases() >>> from pprint import pprint >>> pprint(list(phases.find())) [('SaveOldApplianceAddress', None), ('RestartDhcp', None), ('ResolvConf', None), ('LdapDhcp', None), ('FlushOldAddresses', None), ('LdapSSO', None), ('LdapDns', None), ('RestartListener', None), ('LdapSamba', None), ('LdapSelf', None), ('RestartPostgresql', None), ('RestartCups', None), ('LdapNetwork', None), ('RestartBind', None), ('RewriteWins', None), ('RestartSamba4', None), ('__init__', None), ('RestartKerberos', None), ('LdapReferences', None), ('RewriteUcr', None), ('RestartSamba', None), ('KernelModules', None), ('RestartNscd', None), ('RestartAllInterfaces', None), ('RewritePxe', None)] >>> phases.load() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3/dist-packages/univention/management/console/modules/setup/netconf/modules/__init__.py", line 69, in load module = importlib.util.module_from_spec(info) File "<frozen importlib._bootstrap>", line 580, in module_from_spec AttributeError: 'NoneType' object has no attribute 'loader'
(In reply to Florian Best from comment #10) > The DVD Installation tests are failing due to > d34605ac9ea18112b538782ae65d348d1e742ced. > > >>> from univention.management.console.modules.setup.netconf.modules import RunPhases Who ever investigates this: please fix my logic bug in that code, where in two exception cases the exception is only printed, but the code continues with broken / uninitialized values. Please add the appropriate `continue` statements: diff --git a/base/univention-system-setup/umc/python/setup/netconf/modules/__init__.py b/base/univention-system-setup/umc/python/setup/netconf/modules/__init__.py index ec597da9c7..2b7fd2f15e 100644 --- a/base/univention-system-setup/umc/python/setup/netconf/modules/__init__.py +++ b/base/univention-system-setup/umc/python/setup/netconf/modules/__init__.py @@ -63,6 +63,7 @@ class RunPhases(object): info = importlib.util.find_spec(name, [dirpath]) except ImportError: self.logger.warning("Failed to open '%s'", filename) + continue yield name, info def load(self) -> None: @@ -73,6 +74,7 @@ class RunPhases(object): info.loader.exec_module(module) except SyntaxError as ex: self.logger.warning("Failed to import '%s': %s", name, ex) + continue for key, value in vars(module).items(): if not key.startswith('_'): self.add(key, value)
(In reply to Florian Best from comment #10) > The DVD Installation tests are failing due to > d34605ac9ea18112b538782ae65d348d1e742ced. … > "/usr/lib/python3/dist-packages/univention/management/console/modules/setup/ > netconf/modules/__init__.py", line 69, in load > module = importlib.util.module_from_spec(info) > File "<frozen importlib._bootstrap>", line 580, in module_from_spec > AttributeError: 'NoneType' object has no attribute 'loader' I applied a patch from @fbest and slightly tweaked it by adding my comment 11 [5.0-3] 85baeff0cd fix(USS): Fix netconf module loading base/univention-system-setup/debian/changelog | 6 ++++++ .../umc/python/setup/netconf/modules/__init__.py | 26 ++++++++++---------------- doc/errata/staging/univention-system-setup.yaml | 2 +- 3 files changed, 17 insertions(+), 17 deletions(-) Package: univention-system-setup Version: 13.0.6-4 Branch: ucs_5.0-0 Scope: errata5.0-3
Some commits form Philipp were added as well: 8dbea32c69 fixup! refactor(lib): Code cleanup 09d3d90b1a fixup! refactor(lib): Rename variables for type change 85baeff0cd fix(USS): Fix netconf module loading 35392c706e refactor(AppC): Reorder exception handling 2ef9e367be refactor(AppC): Optimize list element de-duplication e8c6d17158 refactor(AppC): Fix bytes encoding 7224bc865d refactor(AppC): Stop passing around None 78c52b5d3a refactor(AppC): verbose_http_error 074195c239 doc(AppC): More PEP-484 type hinting 4901eb7bbc fix(USS): dhclient code 9bd10c3e17 refactor(USS): dict comprehension 2ec66c6edc refactor(USS): Drop Python 2.x compatibility c1bd9c179c refactor(USS): Code cleanup 71d4111acd refactor(USS): Fix bytes encoding fe00cce78a refactor(USS): Stop passing around None 0256d8c3a0 fix(USS): Remove undefined _*_error variables 53937a004b refactor(USS): Rename variables for type change 9104063d7f doc(USS): More PEP-484 type hinting d58b21b50a style(lib): shellcheck 5a2ccb23d0 doc(lib): Global type declaration 925ecab631 refactor(lib): Ignore stderr faa38d09e9 style(lib): Simplify UCR loading aeb07b074d style(lib): Properly implement TextIOBase protocol 0340844fc3 style(lib): Rename result variable to locked 6e5f6efae4 fix(lib): Always unlock APT bb0f525652 refactor(lib): Code cleanup acece00ffa refactor(lib): Fix bytes encoding 17d815ef98 refactor(lib): Stop passing around None 49331e205b refactor(lib): Rename variables for type change 0d61cc7267 doc(lib): More PEP-484 type hinting
A lot of things from Bug #55915 have also been resolved.
OK: Python 2 references have been removed, where they are obsolete OK: Python 3.9 and Python 3.11 compatibility have been added OK: Python 2.7 compatibility is still existing for components providing this OK: Philipps changes in MR: https://git.knut.univention.de/univention/ucs/-/merge_requests/719 OK: YAML
<https://errata.software-univention.de/#/?erratum=5.0x679> <https://errata.software-univention.de/#/?erratum=5.0x680> <https://errata.software-univention.de/#/?erratum=5.0x681> <https://errata.software-univention.de/#/?erratum=5.0x682> <https://errata.software-univention.de/#/?erratum=5.0x683> <https://errata.software-univention.de/#/?erratum=5.0x684> <https://errata.software-univention.de/#/?erratum=5.0x685> <https://errata.software-univention.de/#/?erratum=5.0x686> <https://errata.software-univention.de/#/?erratum=5.0x687> <https://errata.software-univention.de/#/?erratum=5.0x688> <https://errata.software-univention.de/#/?erratum=5.0x689> <https://errata.software-univention.de/#/?erratum=5.0x690> <https://errata.software-univention.de/#/?erratum=5.0x691> <https://errata.software-univention.de/#/?erratum=5.0x692> <https://errata.software-univention.de/#/?erratum=5.0x693> <https://errata.software-univention.de/#/?erratum=5.0x694> <https://errata.software-univention.de/#/?erratum=5.0x695> <https://errata.software-univention.de/#/?erratum=5.0x696> <https://errata.software-univention.de/#/?erratum=5.0x697> <https://errata.software-univention.de/#/?erratum=5.0x698> <https://errata.software-univention.de/#/?erratum=5.0x699>