python3-setuptools: patch entrypoints for faster initialization

setuptools' pkg_resources module has major performance issues with how
it loads entry points (e.g. the console_script entry point, which sets
up a module as a command-line executable), leading even the simplest
"hello world" scripts to take on the order of 150ms to run if
pkg_resources is incorporated. This is prohibitive for code that needs
to run quickly, and so we patch setuptools to reduce this time. As of
Python 3.7, importlib.resources is available and intended to replace
much of the functionality that causes this sluggishness, but since
many projects still utilize the legacy setuptools modules, a patch is
still required. Note that python3-fastentrypoints (which is available
in the meta-virtualization layer) is also intended to help alleviate
the problem, but since it must be added to existing projects it has
the same disadvantage as resorting to importlib.resources, requiring
manual additions to existing code to see the performance gains.

The intent here is to patch easy_install to load module entry points
directly with the installed setuptools, rather than importing
pkg_resources and having it search out the entry points itself. This
leads to a drastic performance improvement - the changes in this patch
have been shown to result in load time ~6-8x lower, depending on the
complexity of the code it is tested with. A simple "hello world"
example on core-image-full-cmdline gave these results with and without
the patch:

core-image-full-cmdline, without setuptools ScriptWriter patch:

root@qemux86-64:~# time /usr/bin/minimal
hello world

real    0m0.198s
user    0m0.174s
sys     0m0.023s

core-image-full-cmdline, with setuptools ScriptWriter patch:

root@qemux86-64:~# time /usr/bin/minimal
hello world

real    0m0.034s
user    0m0.024s
sys     0m0.010s

More details on the pkg_resources issue are available at:
https://github.com/pypa/setuptools/issues/510

(From OE-Core rev: 9ff7c2f4a43e28ac6a89045c38effe03063f2061)

Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Trevor Gamblin
2020-06-23 19:42:46 -04:00
committed by Richard Purdie
parent e95c36f8a7
commit 5951cbcabe
2 changed files with 65 additions and 1 deletions

View File

@@ -0,0 +1,62 @@
From aae8cd3de3f289cea3db01212579913c925191e8 Mon Sep 17 00:00:00 2001
From: Lauri Tirkkonen <lauri.tirkkonen.ext@nokia.com>
Date: Thu, 26 Mar 2020 14:24:25 +0000
Subject: [PATCH] ScriptWriter: create more efficient /usr/bin wrappers
Upstream setuptools writes scripts to /usr/bin that do insanely much
stuff at runtime. https://github.com/pypa/setuptools/issues/510
Since the script entry points are already known at build time, we can
just write those directly into the /usr/bin wrapper, avoiding the
expensive 'pkg_resources' import at runtime. The idea is from
https://github.com/ninjaaron/fast-entry_points but patched directly into
the native build of setuptools here, so that all Python modules under
bitbake automatically use it without needing additional build time
dependencies.
Upstream-Status: Pending
Signed-off-by: Lauri Tirkkonen <lauri.tirkkonen.ext@nokia.com>
Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
---
setuptools/command/easy_install.py | 14 ++++++--------
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git a/setuptools/command/easy_install.py b/setuptools/command/easy_install.py
index 8fba7b41..03a72714 100755
--- a/setuptools/command/easy_install.py
+++ b/setuptools/command/easy_install.py
@@ -2023,17 +2023,12 @@ class ScriptWriter(object):
"""
template = textwrap.dedent(r"""
- # EASY-INSTALL-ENTRY-SCRIPT: %(spec)r,%(group)r,%(name)r
- __requires__ = %(spec)r
- import re
import sys
- from pkg_resources import load_entry_point
+
+ from %(module)s import %(ep0)s
if __name__ == '__main__':
- sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
- sys.exit(
- load_entry_point(%(spec)r, %(group)r, %(name)r)()
- )
+ sys.exit(%(entrypoint)s())
""").lstrip()
command_spec_class = CommandSpec
@@ -2068,6 +2063,9 @@ class ScriptWriter(object):
for type_ in 'console', 'gui':
group = type_ + '_scripts'
for name, ep in dist.get_entry_map(group).items():
+ module = ep.module_name
+ ep0 = ep.attrs[0]
+ entrypoint = '.'.join(ep.attrs)
cls._ensure_safe_name(name)
script_text = cls.template % locals()
args = cls._get_script_args(type_, name, header, script_text)
--
2.24.1