See previous commit (python2 version) for more info, since mostly
everything applies here as well.
Old manifest file had several issues:
- Its unorganized and hard to read and understand it for an average
human being.
- When a new package needs to be added, the user actually has to modify
the script that creates the manifest, then call the script to create
a new manifest, and then submit a patch for both the script and the
manifest, so its a little convoluted.
- Git complains every single time a patch is submitted to the manifest,
since it violates some of its guidelines.
- It changes or may change with every release of python, its impossible
to know if the required files for a certain package have changed
(it could have more or less dependencies), the only way of doing so
would be to install and test them all one by one on separate individual
images, and even then we wouldnt know if they require less dependencies,
we would just know if an extra dependency is required since it would
complain, lets face it, this isnt feasible.
- The same thing happens for new packages, if someone wants to add a new
package, its dependencies need to be checked manually one by one.
Features/Fixes:
- A new manifest format is used (JSON), easy to read and understand.
This file is parsed by the python recipe and python packages
read from here are passed directly to bitbake during parsing time.
- It provides an automatic manifest creation task (explained on previous
commit), which automagically checks for every package dependencies and
adds them to the new manifest, hence we will have on each package
exactly what that package needs to be run, providing finer granularity.
- Dependencies are also checked automagically for new packages
(explained on previous commit).
This patch has the same features as the python2 version but it differs
in the following ways:
- Python3 handles precompiled bytecode files (*.pyc) differently.
for this reason and since we are cross compiling, wildcards couldnt be
avoided on python3 (See PEP #3147 [1]).
Both the manifest and the manifest creation script handle this
differently, the manifest for python3 has an extra field for cached
files, which is how it lets the user install the cached files or not
via : INCLUDE_PYCS = "1" on their local.conf.
- Shared libraries nomenclature also changed on python3, so again, we
use wildcards to deal with this issue ( See PEP #3149 [2]):
- Fixes python3 manifest, python3-core should be base and everything
should depend on it, hence several packages were deleted:
python3-enum, re, gdbm, subprocess, signal, readline.
- When building python3-native it adds as symlink to it called
nativepython3, which is then isued by the create_manifest task.
- Fixes [YOCTO #11513] while were at it.
References:
[1] https://www.python.org/dev/peps/pep-3147/
[2] https://www.python.org/dev/peps/pep-3149/
(From OE-Core rev: 54ac820b8a639950ccb534dcd9d6eaf8b2b736e0)
Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove two unneeded hacks. The first hack ("setup.py: no host headers
libs" patch) is not needed because we use cross-compiler (e.g.
i586-oe-linux-gcc) which has not been configured with any host system
include or library directories, and thus, we don't get any host system
directories when running "gcc -E -v".
The second hack becomes useless after the first hack has been removed
and we get the standard include and lib directories normally from gcc.
(From OE-Core rev: 9e9545ef818adfb57f428db96d4c2ebb0ea65ec6)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bsddb module was removed in Python 3 and the dbm module doesn't support
Berkeley DB as an option, so this build dependency can be removed.
(From OE-Core rev: f16eac28d909bc6570d3587a2f4ac26ae0dc6048)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The compiled .pyc files contain time stamp corresponding to the compile time.
This prevents binary reproducibility. This patch allows to achieve binary
reproducibility by overriding the build time stamp by the value
exported via SOURCE_DATE_EPOCH.
Patch by Bernhard M. Wiedemann.
[YOCTO#11241]
(From OE-Core rev: 2175aec10a764bfc925a3fb447547982d0ae662f)
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In upstream, the following commit:
e711cafab1
...
commit e711cafab13efc9c1fe6c5cd75826401445eb585
Author: Benjamin Peterson <benjamin@python.org>
Date: Wed Jun 11 16:44:04 2008 +0000
Merged revisions 64104,64117 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk
...
(see diff in setup.py)
It assigned libraries for multiprocessing module according
the host_platform, but not pass it to Extension.
In glibc, the following commit caused two definition of
sem_getvalue are different.
https://sourceware.org/git/?p=glibc.git;a=commit;h=042e1521c794a945edc43b5bfa7e69ad70420524
(see diff in nptl/sem_getvalue.c for detail)
`__new_sem_getvalue' is the latest sem_getvalue@@GLIBC_2.1
and `__old_sem_getvalue' is to compat the old version
sem_getvalue@GLIBC_2.0.
If not explicitly link to library pthread (-lpthread), it will
load glibc's sem_getvalue randomly at runtime.
Such as build python on linux x86_64 host and run the python
on linux x86_32 target. If not link library pthread, it caused
multiprocessing bounded semaphore could not work correctly.
...
>>> import multiprocessing
>>> pool_sema = multiprocessing.BoundedSemaphore(value=1)
>>> pool_sema.acquire()
True
>>> pool_sema.release()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: semaphore or lock released too many times
...
And the semaphore issue also caused multiprocessing.Queue().put() hung.
(From OE-Core rev: ca1542cdf6b6437a2f3dcdb33ac5216bf841c04a)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When py3 applications are exiting we often see errors similar to the
following:
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7fcb56b09400>
Traceback (most recent call last):
File "/usr/lib64/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
After a quick search this was found to be a well reported issue
upstream and had an appropriate fix which is backported here.
(From OE-Core rev: 8aaf09a916a2f66f1a6a79cbddf45390ecefde4f)
Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Porting patch from
<409482251b>
to use _sysconfigdata.py to initialize distutils.sysconfig.
This patch makes that distutils.sysconfig doesn't need config-*/Makefile
in libdir any more. Next we can move it from python3-misc to python3-dev
package.
(From OE-Core rev: 659a80afc5894658f8b82fcd62ebe3562b441db9)
Signed-off-by: Li Zhou <li.zhou@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prior versions of python do not support openssl 1.1; updating to
Python 3.6 on the other hand is a lot more involved, and so should
be done by a specialist/maintainer.
LICENSE checksum change due to copyright years.
Drop upstreamed python3-fix-CVE-2016-1000110.patch
Rebase upstream-random-fixes.patch (taken from
ff558f5aba )
Rebase 0001-Do-not-use-the-shell-version-of-python-config-that-w.patch
Rebase 000-cross-compile.patch
(From OE-Core rev: b7b982a29e5d14c558b5fc25b4dc727810510ade)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
