Commit Graph

93 Commits

Author SHA1 Message Date
Martin Jansa
e5f6e0853e python: add a fix for CVE-2019-9948 and CVE-2019-9636
(From OE-Core rev: 9d23b982fa4e0290761b3d15f6959779fed72ad6)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-21 12:56:34 +01:00
Alexander Kanavin
0ee1b6d0c9 python: update to 2.7.16
Drop backported patches

License-update: copyright years

(From OE-Core rev: 061dfcdf062d64e4e1e50e28edfacb14e41b7d74)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-16 11:10:02 +01:00
Richard Purdie
57933bd958 openssh/util-linux/python*: Ensure ptest output is unbuffered
We need to run sed with the -u option to ensure the output is unbuffered else
ptest-runner may timeout thinkig things were idle. Busybox doesn't have the -u
option so we need to RDEPEND on sed (which is a good thing to do if we use it
anyway).

Alex Kanavin should get credit for discovering the problem.

(From OE-Core rev: d3ffbebf43c23faa43af81c9ecf6fcaef36d675b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09 13:44:39 +01:00
Ross Burton
9dec37ceef python: improve ptest
Add missing runtime dependencies: the test suite needs proper chmod from
coreutils, and the Europe/Minsk timezone.

Also change run-ptest to use -W (run verbosely on failure) instead of -v (always
verbose) to hopefully reduce the noise in the ptest reports.

(From OE-Core rev: 7574f6850797fb7556d2f8077860b7c1d26ae8ec)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-02 15:24:11 +01:00
Alexander Kanavin
d772d8bb1b python: make the python 3.x version of 2to3 utility the default one
(From OE-Core rev: 98ba7ba8c5bc091f5cc590299f1a636c9b8dc1f5)

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-08 10:57:19 +00:00
Richard Purdie
9fe8b85f20 python2-manifest: Add missing xmlrpclib.py
The manifest creation bug that was masking this file was fixed, rerun and add
the missing file to fix:

  File "/usr/lib64/python2.7/SimpleXMLRPCServer.py", line 102, in <module>
    import xmlrpclib
ImportError: No module named xmlrpclib

[YOCTO #12814]

(From OE-Core rev: e203b71b06df1e7d73fa300d45c428008a190d00)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-02 11:05:59 +00:00
Changqing Li
82315924b7 python/python3: use cc_basename to replace CC for checking compiler
When working path contains "clang"/"gcc"/"icc", it might be part of $CC
because of the "--sysroot" parameter. That could cause judgement error
about clang/gcc/icc compilers.

eg: if build under /yocto/builds/xicc/,  bitbake python,  $CC will contains
xicc, will make $CC match *icc, but actuall xicc just folder name.

When "*icc" is matched, below errors are reported when
compiling python/python3:
x86_64-wrs-linux-gcc: error: strict: No such file or directory
x86_64-wrs-linux-gcc: error: unrecognized command line option '-fp-model'

Here use cc_basename to replace CC for checking compiler to avoid such
kind of issue.

(From OE-Core rev: 96383efa2726ed1bf7893332d726112a8552fc24)

Signed-off-by: Li Zhou <li.zhou@windriver.com>

Signed-off-by: Changqing Li <changqing.li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-01-07 14:38:39 +00:00
Alejandro Enedino Hernandez Samaniego
d0c2395db3 create_manifest2: Dont match filenames which contain the directory name for new manifest
When creating a new python2 manifest, there is a corner case on which
the filepath for a certain dependency that was found, could contain
the path of an existing folder, e.g. ${libdir}/python2.7/xmlrpclib.py
module path contains ${libdir}/python2.7/xml, this causes an issue where
the dependency doesnt get eventually added on FILES for that module.

This patch checks if the dependency that was found is a directory, if it
is, it checks if it matches one of the existing directories on the
manifest, if it is not, then it checks if the dependency's path (without
the filename) matches one of the directories.

Also some misc indentation fixes.

(From OE-Core rev: a1c1253b44eb2000de55b7fa3836e5cdaa28a508)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-13 16:32:21 +00:00
Alejandro Enedino Hernandez Samaniego
8627773268 python: Adds instructions to the manifest file
While there is a bit of documentation regarding building a new
manifest file for python, it seems that users usually only read
the manifest file.

The manifest file is in JSON format which doesn't allow comments,
hence why instructions were initially put elsewhere.

This patch hacks the call to open the JSON manifest file by using a
marker to trick it into reading only part of the file as the manifest
itself, and keep the other part as comments, which contain instructions
for the user to run the create_manifest task after an upgrade or
when adding a new package.

(From OE-Core rev: 5641a24a70b54544012c04c6a082514d9a5aa49a)

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-01 11:38:37 +00:00
Mingli Yu
8b7c98e0e0 python: add tk support
Add support to enable tk via PACKGECONFIG.
before the patch:
 # python
 Python 2.7.15 (default, Nov  8 2018, 04:53:50)
 [GCC 8.2.0] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 >>> import Tkinter
 Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/lib-tk/Tkinter.py", line 39, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
 ImportError: No module named _tkinter
 >>>

After the patch, if enable tk in PACKGECONFIG, then
 # python
 Python 2.7.15 (default, Oct 25 2018, 08:12:45)
 [GCC 8.2.0] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 >>> import Tkinter
 >>>

(From OE-Core rev: 26b2702475697c123937d237e00c34eb2216b5ea)

Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-11-23 23:35:19 +00:00
Ross Burton
bc395e0e87 python: add missing CVE tag to patches
(From OE-Core rev: 67f9e9045ab91a9df15876ad73e44ff98f11bf59)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-29 17:01:45 +00:00
Chen Qi
4ad151a53d python: backport patch to fix CVE-2018-14647
Backport patch to fix the following CVE.

CVE: CVE-2018-14647

(From OE-Core rev: 68e51756f67499081c3c53cff6c5c1efdf4b60f0)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-20 22:40:16 +01:00
Chen Qi
cdec724312 python: backport patch to fix CVE-2018-1000802
Backport a patch to fix the following CVE.

CVE: CVE-2018-1000802

(From OE-Core rev: c0343f1035af98cb451eea0de94c16fe89ffdf48)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-20 22:40:16 +01:00
Ross Burton
cd9b883f96 python: don't use runtime checks to identify float endianism
Python uses AC_RUN_IFELSE to determine the byte order for floats and doubles,
and falls back onto "I don't know" if it can't run code.  This results in
crippled floating point numbers in Python, and the regression tests fail.

Instead of running code, take a macro from autoconf-archive which compiles C
with a special double in which has an ASCII representation, and then greps the
binary to identify the format.

This is essentially a backport of the Python 3 patch in oe-core 1781b87.

(From OE-Core rev: 94cea72a23a374eb616d5642977b45172537beac)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-14 09:20:56 +01:00
Ross Burton
64ff0a5e6c python: clean up ptest
As the manifest handling is done differently now, just inherit ptest with the
other inherits.

test_shutil needs unzip so add to RDEPENDS.

Instead of using a patched Makefile, call test.regrtest directly.

(From OE-Core rev: 84f34ad223b1e3f36cab2ac12246eb90efc919bc)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-14 09:20:56 +01:00
Ross Burton
bbe137b05b python: fix patch headers
(From OE-Core rev: 5aba4b7540a71908a5e1086f444b2f205ba5829d)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-08 16:04:12 +01:00
Anuj Mittal
2b4dce4925 python: fix failing ssl tests
Backport two and pick some other in-review patches from Ubuntu to fix
ssl test failures because of OpenSSL 1.1.x changes.

Fixes [YOCTO #12788]

(From OE-Core rev: 4f4bbb936231dd30c3745ef573993f1062937ffd)

Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-08 14:13:54 +01:00
Ross Burton
4612cb45ba python: move sqlite module into python-sqlite
The module was accidentally included in python-misc.

Also re-run create_manifest to update the dependencies of python-sqlite3,
causing some whitespace changes.

[ YOCTO #12933 ]

(From OE-Core rev: 9c2b1a0bfc8783052b5dda344cc334b9c13736f5)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-01 13:04:49 +01:00
Ross Burton
f27fc7f336 python: mark the tests modules as special when updating the manifest
We manually maintain the tests package's content and dependencies, so mark is as
special (matching create_manifest3.py).

(From OE-Core rev: b4bd4e7e26bb79223129abb9fb6e22e9045e3707)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-10-01 13:04:49 +01:00
Andrew Geissler
3c69d1a128 python: consolidate tests
Currently the bulk of the tests in python-tests, some more in
python-sqlite3-tests, and others in their parent module (such as
python-ctypes).  This is pointless space usage if we're not planning on
running the tests, so consolidate all the tests into python-tests.

This is a backport of the same changes done by Ross Burton for python3

Changes since v1:
- Rebase

(From OE-Core rev: 0e94737e7124f689c3697d4227bfcd228cc04295)

Signed-off-by: Andrew Geissler <geissonator@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-21 18:45:46 -07:00
Andrew Geissler
b0f5feea01 python: don't sort the manifest in create_manifest
Instead of sorting the entire manifest when it is updated, use
OrderedDict to preserve the order of fields.This means that
packages can be ordered in the manifest to allow non-trivial
FILES assignments (such as a package that picks up pieces of
other packages)

The manifest has been regenerated with the new stable ordering, and
distutils-staticdev moved above distutils so the packaging rules work as
expected.

This is a backport of the same changes done by Ross Burton for python3

Changes since v1:
- Moved distutils-staticdev above distutils so packaging rules work
as expected.
Changes since v2:
- Rebase

(From OE-Core rev: 3c62c42ebde9dd4acdc74c56160d6ce8639b497c)

Signed-off-by: Andrew Geissler <geissonator@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-09-21 18:45:46 -07:00
Derek Straka
e0cbfaedc4 python: update to version 2.7.15
Update to the latest stable version

License-Update: Copyright year updated to include 2018

Remove the alignment patch that is included upstream

(From OE-Core rev: 855020053906478cea164ed254c08bedce48479d)

Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-05 11:39:06 +01:00
Ross Burton
dfe68bd964 python2: Fix build with gcc8
(From OE-Core rev: 910f68c9c8dc26e12d28ef29e956af63d100f121)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-22 13:13:33 +01:00
Koen Kooi
f03dccc7ec python 2.7: fix multilib patch to accept multiarch style paths
Using 'basename' to strip the prefix fails when using multiarch style paths.

(From OE-Core rev: c61c416a6504f7e8885df3c94c839d1031920a1c)

Signed-off-by: Koen Kooi <koen.kooi@linaro.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-05-04 13:28:04 +01:00
Ross Burton
a2fdf7fb08 python: refresh patches
The patch tool will apply patches by default with "fuzz", which is where if the
hunk context isn't present but what is there is close enough, it will force the
patch in.

Whilst this is useful when there's just whitespace changes, when applied to
source it is possible for a patch applied with fuzz to produce broken code which
still compiles (see #10450).  This is obviously bad.

We'd like to eventually have do_patch() rejecting any fuzz on these grounds. For
that to be realistic the existing patches with fuzz need to be rebased and
reviewed.

(From OE-Core rev: 8a5c1328c4ea63443a92813c54bd2229c9959ff9)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-11 06:27:00 -07:00
Ross Burton
fe9e4bbee7 python: refresh patches
The patch tool will apply patches by default with "fuzz", which is where if the
hunk context isn't present but what is there is close enough, it will force the
patch in.

Whilst this is useful when there's just whitespace changes, when applied to
source it is possible for a patch applied with fuzz to produce broken code which
still compiles (see #10450).  This is obviously bad.

We'd like to eventually have do_patch() rejecting any fuzz on these grounds. For
that to be realistic the existing patches with fuzz need to be rebased and
reviewed.

(From OE-Core rev: 0b25fcee333e6207a8596d26adfa65fec85c26df)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-09 09:17:04 -08:00
Jason Wessel
0fa0baed74 python3: Fix install purelib to make pip3-python work properly
The oe-core version of python3 patches the purelib use directory to
the system libdir so as to make it work with multilibs properly inside
the patch fix_for_using_different_libdir.patch with:

-        'purelib': '{base}/lib/python{py_version_short}/site-packages',
+        'purelib': '{base}/'+sys.lib+'/python{py_version_short}/site-packages',

The problem is that this broke the pip3-python package because the
install directory is out of sync when using a multilib version of
python.  When ever a module is installed with pip3 install that is a
purelib it will get installed to a location that python3 will never
reference and cause random failures.

This patch fixes the purelib install directory to match the purelib
use directory for externally managed python modules when using
multilibs.

(From OE-Core rev: 45afadf0b652922f9e60c5a778acd3612da83306)

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-03-08 10:39:33 -08:00
Derek Straka
98c4f485f0 python2-manifest: update the manifest file for python 2.7.14
The manifest file was not regenerated when the package was updated.

(From OE-Core rev: c42a1cf42b88c0dc02baf807ee2aa3dab058c4cd)

Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-06 11:06:27 +00:00
Derek Straka
7ef064e80c python: Upgrade both python and python-native to 2.7.14
Rebased:
  - python/01-use-proper-tools-for-cross-build.patch
  - python/fix-makefile-for-ptest.patch
  - python/parallel-makeinst-create-bindir.patch

Removed Upstreamed Patch:
  - python/Don-t-use-getentropy-on-Linux.patch

Updated license checksum for changes in the copyright date.  The license
terms remain unchanged

Added an extra do_compile item to create the native pgen that no longer
gets compiled by default

(From OE-Core rev: 9f2de4f9cf1eb6de75dc789bd0549f45c7a68c55)

Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-29 08:49:50 +00:00
Tim Orling
5dd24c1753 python*-manifest.json: add dependencies, runpy
* runpy allows running modules/scripts with 'python -m foo'
  - create explicit python2 and 3 packages rather than the
    misc catchall
* python3-setuptools and html.parser RDEPENDS on _markupbase
  - add to python3-core rather than misc catchall
* pip3 RDEPENDS on plistlib, http.client
  - already packaged in python2, add to python3
  - add http/ to -netclient
* "pip3 install" RDEPENDS on encodingds.idna
  - encodings.idna packaged in -core, but missing:
    - stringprep (move from -codecs to -core)
    - unicodedata (move from -codecs to -core)

(From OE-Core rev: 65a85c7db3de8d16ff91a5208a59cc2202d34e5b)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-20 22:31:56 +00:00
Tim Orling
86e002572d python2 create_manifest.py: fix trailing whitespace in json
The json.dumps function adds trailing whitespace when using
indent, because the default separator is not ','.

The workaround [1] is to set the separators to be ',' and ': ',
e.g. separators=(',', ': ')

[1] https://hg.python.org/cpython/rev/78bad589f205

(From OE-Core rev: e4cc287e41af2e52240a20d4bf2b6cc0f1f85314)

Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-20 22:31:56 +00:00
Alejandro Hernandez
8d94b9db22 python: Restructure python packaging and replace it with autopackaging
The reason we have a manifest file for python is that our goal is to
keep python-core as small as posible and add other python packages only
when the user needs them, hence why we split upstream python into several
packages.

Although our manifest file has several issues:
 - Its unorganized and hard to read and understand it for an average
   human being.
 - When a new package needs to be added, the user actually has to modify
   the script that creates the manifest, then call the script to create
   a new manifest, and then submit a patch for both the script and the
   manifest, so its a little convoluted.
 - Git complains every single time a patch is submitted to the manifest,
   since it violates some of its guidelines.
 - It changes or may change with every release of python, its impossible
   to know if the required files for a certain package have changed
   (it could have more or less dependencies), the only way of doing so
   would be to install and test them all one by one on separate individual
   images, and even then we wouldnt know if they require less dependencies,
   we would just know if an extra dependency is required since it would
   complain, lets face it, this isnt feasible.
 - The same thing happens for new packages, if someone wants to add a
   new package, its dependencies need to be checked manually one by one.

This patch fixes those issues, while adding some additional features.

Features/Fixes:

 - A new manifest format is used (JSON), easy to read and understand.
   This file is parsed by the python recipe and python packages read
   from here are passed directly to bitbake during parsing time.
 - It provides an automatic manifest creation task (explained below),
   which automagically checks for every package dependencies and adds
   them to the new manifest, hence we will have on each package exactly
   what that package needs to be run, providing finer granularity.
 - Dependencies are also checked automagically for new packages (explained below).
 - Fixes the manifest in the following ways:
   * python-core should be base and all packages should depend on it,
     fixes lang, string, codecs, etc.
   * Fixes packages with repeated files (e.g. bssdb and db, or
     netclient and mime, and many others).
 - Sitecustomize was fixed since encoding was deprecated.
 - The JSON manifest file invalidates bitbake's cache, so if it changes
   the python package will be rebuilt.
 - It creates a solution for users that want precompiled bytecode files
   (*.pyc) INCLUDE_PYCS = "1" can be set by the user on their local.conf to
   include such files, some argument they get faster boot time, even when the
   files would be created on their first run?, but they also sometimes give a
   magic number error and take up space, so we leave it to the user to
   decide if they want them or not.
 - Fixes python-core dependencies, e.g.
   When python is run on an image, it TRIES to import everything it needs,
   but it doesnt necessarily fails when it doesnt find something, so even if
   we didnt know, we had errors like (trimmed on purpose):
   # trying /usr/lib/python2.7/_locale.so
   # trying /usr/lib/python2.7/lib-dynload/_locale.so
   # trying /usr/lib/python2.7/_sysconfigdata.so

   while it didnt complain about _locale it should have imported it,
   after creating a new manifest with the automated script we get:

   # trying /usr/lib/python2.7/lib-dynload/_locale.so
   dlopen("/usr/lib/python2.7/lib-dynload/_locale.so", 2);
   import _locale # dynamically loaded from /usr/lib/python2.7/lib-dynload/_locale.so

How to use (after a new release of python, or maybe before every OE
release):
 - A new task called create_manifest was added to the python package,
   which may be invoked via:

 $ bitbake python -c create_manifest

 This task runs a script on native python on our HOST system, and since
 the python and python-native packages come from the same source, we can
 use it to know the dependencies of each module as if we were doing it
 on an image, this script is called create_manifest.py and in a very
 simplistic way it does the following:

 1. Reads the JSON manifest file and creates a dictionary data structure
    with all of our python packages, their FILES, RDEPENDS and SUMMARY.
 2. Loops through all of them and runs every module listed on them
    asynchronously, determining every dependency that they have.
 3. These module dependencies are then handled, to be able to know which
    packages contain those files and which should RDEPEND on one another.
 4. The data structure that comes out of this, is then used to create a
    new manifest file which is automatically copied onto the user's python
    directory replacing the old one.

 Create_manifest script features:
 - Handles modules which dont exist anymore (new release for example).
 - Handles modules that are builtin.
 - Deals with modules which were not compiled (e.g. bsddb or ossaudiodev)
 - Deals with packages which include folders.
 - Deals with packages which include FILES with a wildcard.
 - The manifest can be constructed on a multilib environment as well.
 - This method works for both python modules and shared libraries used
   by python.

How to add a new package:
 - If a user wants to add a new package all that has to be done is
   modify the python2-manifest.json file, and add the required file(s)
   to the FILES list, the script should handle all the rest.
   Real example:
   We want to add a web browser package, including the file webbrowser.py
   which at the moment is on python-misc.
   "webbrowser": {
       "files": ["${libdir}/python2.7/lib-dynload/webbrowser.py"],
       "rdepends": [],
       "summary": "Python Web Browser support"}

 Run bitbake python -c create_manifest and the resulting manifest
 should  be completed after a few seconds, showing something like:
   "webbrowser": {
      "files": ["${libdir}/python2.7/webbrowser.py"],
      "rdepends": ["core","fcntl","io","pickle","shell","subprocess"],
      "summary": "Python Web Browser support"}

Known errors/issues:
 - Some special packages are handled differently: core, misc,
   modules,dev, staticdev.
   All these should be handled manually, because they either include
   binaries, static libraries, include files, etc. (something that we
   cant import).
   Specifically static libraries are not not supported by this method
   and have to be handled by the user.
 - The change should be transparent to the user, other than the fact
   that now we CANT build python-foo (it was pretty dumb anyway, since
   what building python-foo actually did was building the whole python
   package anyway), but doing IMAGE_INSTALL_append = " python-foo"
   would create an image with the requested package with no issues.

[YOCTO #11510] [YOCTO #11694] [YOCTO #11695]

(From OE-Core rev: 6959e2e4dba5bbfa6ffd49c44e738cc1c38bc280)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-01-20 22:31:56 +00:00
Juro Bystricky
07348bb76f python2.7: improve reproducibility
The compiled .pyc files contain time stamp corresponding to the compile time.
This prevents binary reproducibility. This patch allows to achieve binary
reproducibility by overriding the build time stamp by the value
exported via SOURCE_DATE_EPOCH.

Patch by Bernhard M. Wiedemann, backported from https://github.com/python/cpython/pull/296

[YOCTO#11241]

(From OE-Core rev: 2a044f1e4f5c63e11e631b31f741c7aabfa6f601)

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-23 12:06:51 +01:00
Hongxu Jia
365b18dbef python2/python3: fix multiprocessing.BoundedSemaphore not work on qemux86/qemuarm
In upstream, the following commit:
e711cafab1
...
commit e711cafab13efc9c1fe6c5cd75826401445eb585
Author: Benjamin Peterson <benjamin@python.org>
Date:   Wed Jun 11 16:44:04 2008 +0000

    Merged revisions 64104,64117 via svnmerge from
    svn+ssh://pythondev@svn.python.org/python/trunk
...
(see diff in setup.py)
It assigned libraries for multiprocessing module according
the host_platform, but not pass it to Extension.

In glibc, the following commit caused two definition of
sem_getvalue are different.
https://sourceware.org/git/?p=glibc.git;a=commit;h=042e1521c794a945edc43b5bfa7e69ad70420524
(see diff in nptl/sem_getvalue.c for detail)
`__new_sem_getvalue' is the latest sem_getvalue@@GLIBC_2.1
and `__old_sem_getvalue' is to compat the old version
sem_getvalue@GLIBC_2.0.

If not explicitly link to library pthread (-lpthread), it will
load glibc's sem_getvalue randomly at runtime.

Such as build python on linux x86_64 host and run the python
on linux x86_32 target. If not link library pthread, it caused
multiprocessing bounded semaphore could not work correctly.
...
>>> import multiprocessing
>>> pool_sema = multiprocessing.BoundedSemaphore(value=1)
>>> pool_sema.acquire()
True
>>> pool_sema.release()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: semaphore or lock released too many times
...

And the semaphore issue also caused multiprocessing.Queue().put() hung.

(From OE-Core rev: ca1542cdf6b6437a2f3dcdb33ac5216bf841c04a)

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-08-13 09:27:37 +01:00
Richard Purdie
a5bf271c7c meta: Add/fix missing Upstream-Status to patches
This adds or fixes the Upstream-Status for all remaining patches missing it
in OE-Core.

(From OE-Core rev: 563cab8e823c3fde8ae4785ceaf4d68a5d3e25df)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-27 10:38:43 +01:00
Jussi Kukkonen
82311dcf1f python: Remove unused patches
CVE-2016-5636.patch and avoid_parallel_make_races_on_pgen.patch were
removed from SRC_URI as handled upstream in adf4266524.

(From OE-Core rev: 53c0d1f18d4a11b0130e54466c91320acf4f5b18)

Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-16 22:11:32 +00:00
Alejandro Hernandez
30b2044de6 python: Upgrade both python and python-native to 2.7.13
Rebased:
- python-native/multilib.patch
- python/multilib.patch
- python/01-use-proper-tools-for-cross-build.patch

Upstream:
- CVE-2016-1000110

(From OE-Core rev: 2eaadc5464e3340359b626026d80afb6bc01d3f1)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01 23:27:06 +00:00
Andreas Oberritter
4dfa203a65 python: Don't use getentropy on Linux
Backport a patch from 2.7 branch to fix a regression with glibc
2.24 causing "OSError: [Errno 38] Function not implemented" when
calling urandom() with older kernels.

(From OE-Core rev: 3f2be1c857a44030478ce25b4a722667b73de446)

Signed-off-by: Andreas Oberritter <obi@opendreambox.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-02-15 20:06:41 -08:00
Mingli Yu
fd4391d8ba python: fix CVE-2016-1000110
Backport patch to fix CVE-2016-1000110 from python upstream:
for python2.7
https://hg.python.org/cpython/rev/ba915d561667/
for python3
https://hg.python.org/cpython/rev/a0ac52ed8f79

(From OE-Core rev: 1dd22b9d35983f35c481a1fcf67425aa0fd07a5b)

Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-28 10:16:03 +01:00
Alejandro Hernandez
adf4266524 python: upgrade from 2.7.11 to 2.7.12
LICENSE did not change, only dates were changed

Rebases:
 - multilib.patch
 - 01-use-proper-tools-for-cross-build.patch

Upstream:
 - avoid_parallel_make_races_on_pgen.patch
 - CVE-2016-5636.patch

(From OE-Core rev: 2e64fdc99a0e00bd0a4b4bf09a128e56fd8e9f8b)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-10 10:46:30 +01:00
Armin Kuster
81106cd333 python2: Security fix CVE-2016-5636
Affects python2 < 2.7.11
Base score (4.4) Medium

(From OE-Core rev: 4d1f651047a045955b436357753c7e094468b4ed)

Signed-off-by: Armin Kuster <akuster@mvista.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20 10:28:52 +01:00
Ross Burton
e49d337e84 meta: update patch metadata
Enforce the correct tag names across all of oe-core for consistency.

(From OE-Core rev: 606a43dc38a00cc243f933722db657aea4129f8e)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:24 +01:00
Alexander Kanavin
4977a0743b python-native, python3-native: remove the use of exported HOST_SYS and BUILD_SYS variables
The code that utilized them was superseded by the code (in the same patch!)
that is utilizing STAGING_LIBDIR/STAGING_INCDIR, and wasn't correct in the
first place as HOST_SYS is not necessarily the same as the sysroot directory
name.

(From OE-Core rev: 8834e81a38c24a066bb4fefa93da61011d0db244)

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-01 12:38:41 +01:00
Martin Jansa
258cfa891e python(3): Disable tkinter
* restore changes from:
  http://git.openembedded.org/openembedded-core/commit/?id=53ae544cfdac22c82af452b8c7ebe6664296bd9b
* which were shamelessly removed in upgrade to 2.7.9:
  http://git.openembedded.org/openembedded-core/commit/?id=d4ad95f0d5f08891637c644e85b09da9c4585059
  and then spread to python3 as well
* fixes following issues reported by test-dependencies
  WARN: python3: python3-tkinter rdepends on glibc, but it isn't a build dependency?
  WARN: python3: python3-tkinter rdepends on libpython3, but it isn't a build dependency?
  WARN: python3: python3-tkinter rdepends on tcl-lib, but it isn't a build dependency?

  WARN: python: python-tkinter rdepends on glibc, but it isn't a build dependency?
  WARN: python: python-tkinter rdepends on libpython2, but it isn't a build dependency?
  WARN: python: python-tkinter rdepends on tcl-lib, but it isn't a build dependency?

  and following QA warnings in normal builds:
  python-2.7.11: python-tkinter rdepends on tcl-lib, but it isn't a build dependency, missing tcl in DEPENDS or PACKAGECONFIG? [build-deps]
  python-2.7.11: /usr/lib/python2.7/lib-dynload/_tkinter.so contained in package python-tkinter requires libtk8.6.so, but no providers found in RDEPENDS_python-tkinter? [file-rdeps]
  python3-3.5.1: python3-tkinter rdepends on tcl-lib, but it isn't a build dependency, missing tcl in DEPENDS or PACKAGECONFIG? [build-deps]
  python3-3.5.1: /usr/lib/python3.5/lib-dynload/_tkinter.cpython-35m-arm-linux-gnueabi.so contained in package python3-tkinter requires libtk8.6.so, but no providers found in RDEPENDS_python3-tkinter? [file-rdeps]

(From OE-Core rev: 8cc5e36cd7978186d5ea1fe8af2f2bd6b3df8885)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-25 10:29:14 +00:00
Alejandro Hernandez
40738af46b python: Upgrade 2.7.9 > 2.7.11
- no license change, just dates

Rebased:
- check-if-target-is-64b-not-host.patch
- add-CROSSPYTHONPATH-for-PYTHON_FOR_BUILD.patch

(From OE-Core rev: 9ed4ef038a4a8140accfa97b2eb6b75f8bed3693)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-20 17:07:14 +00:00
Jackie Huang
a1fa8d9ac5 python: Fix cross compiling issue
If the target and host have the same type, the system
may try to execute the instructions from the target
version.  This can lead to illegal instructions
as well as the wrong copy of the code running.

Add CROSSPYTHONPATH for PYTHON_FOR_BUILD and export
the correct path to fix it.

(From OE-Core rev: 1cda7afd4173100e4185ee8759f6b2770bc93b72)

Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-12 23:42:52 +00:00
Ross Burton
1c914a844b meta: Fix Upstream-Status statements
Fix a variety of problems such as typos, bad punctuations, or incorrect
Upstream-Status values.

(From OE-Core rev: bd220fe6ce8c3a0805f13a14706d3130ea872604)

Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-12 23:01:53 +01:00
Alejandro Hernandez
65a8408155 python: Fix PGEN linking errors
Parallel make races when linking PGEN affects target's python
compilation as well, adds patch from python-native to modify the
Makefile and avoid parallel make races, also updates upstream status

(From OE-Core rev: 4f0d1015aa3331ae7b376f836d3188b6017abfb1)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-25 12:39:39 +00:00
Alejandro Hernandez
cb02ecb507 Python: Upgrade from 2.7.3 to 2.7.9:
- Based on Paul Eggletons work to partially upgrade to Python 2.7.6

Modified:
default-versions.inc: switched to python 2.7.9
generate-manifest-2.7.py: fixed _sysconfigdata
python-2.7-manifest.inc: fixed _sysconfigdata
python.inc: Updated checksums and source, no LICENSE
change just updated some dates
python-native_2.7.3 -> python-native_2.7.9 and updated patches
python_2.7.3 -> python_2.7.9, and added ac_cv_file__dev_ptmx=no
ac_cv_file__dev_ptc=no in EXTRA_OECONF to solve python
issue #3754, only needed when cross compiling, also updated patches
use_sysroot_ncurses_instead_of_host.patch: New patch to use ncursesw
from sysroot instead of hosts, introduced by fix for python issue #15268

Rebased:
01-use-proper-tools-for-cross-build.patch
03-fix-tkinter-detection.patch
05-enable-ctypes-cross-build.patch
06-avoid_usr_lib_termcap_path_in_linking.patch
avoid_warning_about_tkinter.patch
builddir.patch
fix_for_using_different_libdir.patch
host_include_contamination.patch
multilib.patch
nohostlibs.patch
search_db_h_in_inc_dirs_and_avoid_warning.patch

Deleted (fixed on upstream):
06-ctypes-libffi-fix-configure.patch
CVE-2013-4073_py27.patch
gcc-4.8-fix-configure-Wformat.patch
json-flaw-fix.patch
posix_close.patch
pypirc-secure.patch
python-2.7.3-CVE-2012-2135.patch
python-2.7.3-CVE-2013-1752-smtplib-fix.patch
python-2.7.3-CVE-2014-1912.patch
python-2.7.3-CVE-2014-7185.patch
python-2.7.3-berkeley-db-5.3.patch
python-fix-build-error-with-Readline-6.3.patch
remove-BOM-insection-code.patch
remove_sqlite_rpath.patch
python2.7.3-nossl3.patch

[YOCTO #7059]

(From OE-Core rev: d4ad95f0d5f08891637c644e85b09da9c4585059)

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-02-23 17:35:29 +00:00
Sona Sarmadi
103663075e python: Disables SSLv3
This is related to "SSLv3 POODLE vulnerability" CVE-2014-3566

Building python without SSLv3 support when openssl is built without
any support for SSLv3 (e.g. by adding EXTRA_OECONF = " -no-ssl3" in
the openssl recipes).

Backport from:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=768611#22
[python2.7-nossl3.patch] only Modules/_ssl.c is backported.

References:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=7015
https://bugzilla.yoctoproject.org/show_bug.cgi?id=6843
http://bugs.python.org/issue22638
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-3566

(From OE-Core rev: 3462cac82cf0ab32e5e530f543b14fdcc211c678)

Signed-off-by: Sona Sarmadi <sona.sarmadi@enea.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-02-02 14:09:20 +00:00