Commit Graph

1228 Commits

Author SHA1 Message Date
Philip Lorenz
d37ef8925a ipk: Fix clean up of extracted IPK payload
It turns out that the IPK payload tarball was actually cleaned up in the
concrete package manager implementation (most likely because at some
point Debian and IPK packages used different compression algorithms).

Globbing removes this ambiguity so move the removal of the payload into
the common extract method.

(From OE-Core rev: ec1c6bc79f5ca219e77a379bbabc2dad808d2020)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1e2b02a54f482159e21902eeb997b21e00e9588e)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-06-05 05:57:12 -07:00
Philip Lorenz
fb4eb8a09d lib/package_manager/ipk: Do not hardcode payload compression algorithm
The chosen payload compression algorithm can be changed by overriding
`OPKGBUILDCMD`. Ensure that package extraction deals with this by
globbing for "data.tar.*" to select the actual payload tarball.

(From OE-Core rev: c2b00cd4148d4b08ba5673488a652f8daf3ac085)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2ad05635a6da403b4fadcc126fe7734067c12c73)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-06-05 05:57:12 -07:00
Ross Burton
0fe72b1a67 lib/oe/package-manager: allow including self in create_packages_dir
This function is typically used to construct a limited feed for image
creation, but there are other cases when you might want a limited feed
and include the current recipe's packages in it.

To ensure that existing behaviour is preserved, add a boolean to control
this behaviour and default it to False.

(From OE-Core rev: 20a6f55328733ad6f0c05b1353e8d525019aeea7)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit aada7fda2b118152d82b1ab295d92b8251afe4ac)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
2024-05-23 08:55:01 -07:00
Jose Quaresma
110ee701b3 sstatesig: fix netrc.NetrcParseError exception
Looks like sometimes the e.filename and the e.lineno is not properly set by the netrc
and this can cause TypeError.

| File "/poky/meta/lib/oe/sstatesig.py", line 342, in init_rundepcheck
|     bb.warn("Error parsing %s:%d: %s" % (e.filename, e.lineno, e.msg))
|             ~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| TypeError: %d format: a real number is required, not NoneType

(From OE-Core rev: 486ed69939f1b7572f121960c2dde246032686e7)

Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-04-04 21:50:24 +01:00
lixiaoyong
66f2f710e6 oe/package: enhance objdump command call with llvm
Replace `${HOST_PREFIX}objdump` with `${OBJDUMP}`.

When utilizing llvm for compiling packages, the invocation of GNU objdump
will consistently occur during package precess. This behavior is unfriendly
to llvm. So prefer `${OBJDUMP}` over `${HOST_PREFIX}objdump`.

(From OE-Core rev: fac4d33041d8feb041e617b9b16689c4d3a522cb)

Signed-off-by: lixiaoyong <lixiaoyong19@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-30 22:25:43 +00:00
Jermain Horsman
9b839a835c bblayers/makesetup.py: Move git utility functions to oe.buildcfg module
This allows other classes to make use of these as well.

Includes a git describe and git toplevel function and functions
to get info for git remotes.

(From OE-Core rev: a04a084b6e513d15cb57ee103c6d6215ce1c75b9)

Signed-off-by: Jermain Horsman <jermain.horsman@nedap.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-30 22:25:43 +00:00
Joshua Watt
df60c6d3ee sstatesig: Warn on bad .netrc
If there is an error parsing .netrc, warn the user

(From OE-Core rev: 47ea9418dfa31a87bc0014c482be5bfdc9df98e1)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-26 17:10:51 +00:00
Joshua Watt
84cb5d2568 sstatesig: Set hash server credentials from bitbake variables
Allows the hash server credentials to be specified in bitbake variables.
If omitted, the users .netrc will be checked

(From OE-Core rev: ba391d39f2b888706e53028e9df3a37c5baedfc1)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-03-23 10:18:20 +00:00
Jonathan GUILLOT
ca25926d07 lib/oe/package: fix LOCALE_PATHS scan to create locale packages
split_locales() must only check subdirectories in paths added to
LOCALE_PATHS to avoid creating weird packages based on filenames also
present in paths.

Without such a filter, cups recipe adding ${datadir}/cups/templates to
LOCALE_PATHS creates the following incorrect packages:
- cups-locale-add-class.tmpl
- cups-locale-add-printer.tmpl
- cups-locale-admin.tmpl

(From OE-Core rev: ba3aee0d516bd066829d6edaa8d7bacdd75dd6ef)

Signed-off-by: Jonathan GUILLOT <jonathan@joggee.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-24 16:10:22 +00:00
Peter Kjellerstedt
4cfd0f7e4e lib/oe/patch: Use git notes to store the filenames for the patches
The old way of keeping track of the filenames for the patches that
correspond to the commits was to add a special comment line to the end
of the commit message, e.g., "%% original patch: <filename>", using a
temporary git hook. This method had some drawbacks, e.g.:

* It caused problems if one wanted to push the commits upstream as the
  comment line had to be manually removed.
* The comment line would end up in patches if someone used git
  format-path rather than devtool finish to generate the patches.
* The comment line could interfere with global Git hooks used to
  validate the format of the Git commit message.
* When regenerating patches with `devtool finish --force-patch-refresh`,
  the process typically resulted in adding empty lines to the end of the
  commit messages in the updated patches.

A better way of keeping track of the patch filenames is to use Git
notes. This way the commit messages remain unaffected, but the
information is still shown when, e.g., doing `git log`. A special Git
notes space, refs/notes/devtool, is used to not intefere with the
default Git notes. It is configured to be shown in, e.g., `git log` and
to survive rewrites (i.e., `git commit --amend` and `git rebase`).

Since there is no longer any need for a temporary Git hook, the code
that manipulated the .git/hooks directory has also been removed. To
avoid potential problems due to global Git hooks, --no-verify was added
to the `git commit` command.

To not cause troubles for those who have done `devtool modify` for a
recipe with the old solution and then do `devtool finish` with the new
solution, the code will fall back to look for the old strings in the
commit message if no Git note can be found.

While not technically motivated like above, the way to keep track of
ignored commits is also changed to use Git notes to avoid having
different methods to store similar information.

(From OE-Core rev: f5e6183b9557477bef74024a587de0bfcc2b7c0d)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 16:03:22 +00:00
Peter Kjellerstedt
ff63bc403d lib/oe/patch: Add GitApplyTree.commitIgnored()
This function can be used to create a commit that devtool will ignore
when creating/updating the patches.

(From OE-Core rev: 94f0838b9223b7ece7affaa707e54a5d784da25e)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 16:03:22 +00:00
Peter Kjellerstedt
24433ce8f9 lib/oe/patch: Make extractPatches() not extract ignored commits
If a commit is marked with "%% ignore" it means it is used by devtool to
keep track of changes to the source code that are not the result of
running do_patch(). These changes need to actually be ignored when
extracting the patches as they typically make no sense as actual patches
in a recipe.

This also adds a new test for oe-selftest that verifies that there are
no patches generated from ignored commits.

(From OE-Core rev: c3d43de7e54189bf09fbe8e87ddb976e42ebf531)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 16:03:22 +00:00
Joshua Watt
ba68f3132d sstatesig: Implement new siggen API
Implements the new API required for querying unihashes in parallel

(From OE-Core rev: 0c66d07238d20dc1b34355e4c472aa7457d2e5fe)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-19 11:58:12 +00:00
Jonathan GUILLOT
c2a7c008e1 lib/oe/package: add LOCALE_PATHS to add define all locations for locales
Some packages may contain localized files not located in default path
${datadir}/locale. Add the new variable LOCALE_PATHS to allow a recipe
to define extra paths or even fully override the scanned directories.
LOCALE_PATHS is set at ${datadir}/locale by default to keep the exact
same behavior for the recipes which did not need modification.

(From OE-Core rev: 0ffc7cf01225743789ac30dd325fca05b9203be1)

Signed-off-by: Jonathan GUILLOT <jonathan@joggee.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-14 13:53:36 +00:00
Jonathan GUILLOT
c381ca7243 lib/oe/package: replace in place PN-locale-* packages in PACKAGES
split_locales() removes PN-locale from PACKAGES and adds PN-locale-* to the end.
As the PN-locale package typically appears before PN base package, it may result
in paths not installed in PN-locale-* packages if already catched by PN. Now
insert PN-locale-* exactly where PN-locale was existing in list to avoid such
an issue.

(From OE-Core rev: 108bc167ed0d43505af3140947a0ab79c89f0a7b)

Signed-off-by: Jonathan GUILLOT <jonathan@joggee.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-14 13:53:36 +00:00
Philip Lorenz
87798f82db ipk: Remove temporary package lists during SDK creation
The temporary package lists used during SDK creation should not be
shipped as part of the SDK (in particular because due to the opkg local
file download optimization they are actually symlinks into the build
directory). Remove them by calling the respective helper method during
the SDK build.

(From OE-Core rev: c18ba66da3c77f247170efd2cb350686010bef57)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-13 13:51:41 +00:00
Martin Jansa
77f5f99fcf package_manager: ipk: add OPKG_MAKE_INDEX_EXTRA_PARAMS variable
* can be used to pass e.g. -f param to preserve user-defined fields
  in the index as added in:
  https://git.yoctoproject.org/opkg-utils/commit/opkg-make-index?id=13f6281d24e17199e0fef6c2984419372ea0f86f

* otherwise it will show a lot of messages like:
  "Lost field Author <value>"
  for every package in the feed

(From OE-Core rev: 6dc772fd9d5c2d90a6e32cfa4bc46ca0221f1b7f)

Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-02-08 10:53:13 +00:00
Ross Burton
bfaea5f7ec cve_check: cleanup logging
Primarily list the number of patches found, useful when debugging.

Also clean up some bad escaping that caused warnings and use
re.IGNORECASE instead of manually doing case-insenstive rang matches.

(From OE-Core rev: 10acc75b7f3387b968bacd51aade6a8dc11a463f)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-23 11:53:41 +00:00
Ross Burton
ab375ea3fe cve_check: handle CVE_STATUS being set to the empty string
Handle CVE_STATUS[...] being set to an empty string just as if it was
not set at all.

This is needed for evaluated CVE_STATUS values to work, i.e. when
setting not-applicable-config if a PACKAGECONFIG is disabled.

(From OE-Core rev: 2c9f20f746251505d9d09262600199ffa87731a2)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-23 11:53:41 +00:00
Richard Purdie
2ebe7f60fd reproducible: Fix race with externalsrc/devtool over lockfile
We occasionally see races over the lockfile used by externalsrc/devtool
when walking files for the source_date_epock calculation. Skip this file
if present to avoid the issues and fix a real issue where SDE could be
contaminated too.

[YOCTO #14921]

(From OE-Core rev: 4bc0eb4bd90e6e6e46581a8ed367212bdd910a26)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-19 12:21:22 +00:00
Richard Purdie
bc883e4991 rootfs-postcommands: Try and improve ordering constraints
The current code is in race to see who can set things last. This isn't scalable or
sustainable and problemtic in the face of inherit ordering changes.

Move the ordering issue into the actual code execution, which isn't ideal but
the best of several bad options and at least lets us drop the anonymous python.

(From OE-Core rev: 0ffff2c1f80a9b79b133d787764bab164d9abd70)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-18 10:15:58 +00:00
Richard Purdie
12619deabd rootfs: Fix MULTILIB_RE_ALLOW to be inherit order independent
This variable is only used by the ipk backend with multilibs.

In order to make it work correctly regardless of inherit order, change
the string to be space delimeted, set using += and add in the regex '|'
sperator at the end of processing.

(From OE-Core rev: 72befdb12568fbc642022ef0a23b269c5b37a638)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-18 10:15:58 +00:00
Etienne Cordonnier
5031cf42ff package.py: fix Darwin support
- 'subprocess.Popen([d.expand("${HOST_PREFIX}otool)' requires text-mode (a more
  readable alias for the universal_newlines parameter), since otool produces
  text and the code 'out.split("\n")' expects a string, not a bytes object.
  otool is used on MacOS only, so this error isn't triggered on Linux.

- use 'startswith("darwin")' in order to support all darwin versions and not
  just specific versions (meta-darwin supports darwin21 at the moment).

(From OE-Core rev: 248ca79a6400e063c4965f9542c614bf837ff758)

Signed-off-by: Dominik Schnitzer <dominik@snap.com>
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-15 21:42:15 +00:00
Yang Xu
b11e3aa995 rootfs.py: check depmodwrapper execution result
The execution result of depmodwrapper is not checked which makes depmod
generation failed sliently and hard to detect.

So check exection result and stop building if depmodwrapper failed.

(From OE-Core rev: 2f88e7d331390c6aaecc4522253e24791aec299e)

Signed-off-by: Yang Xu <yang.xu@mediatek.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-10 17:01:28 +00:00
Alexander Kanavin
4675bbb757 lib/sstatesig/find_siginfo: raise an error instead of returning None when obtaining mtime
Suppressing the error and returning None can result in a delayed failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6254/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6262/steps/14/logs/stdio

It is not clear why the os.stat() error occurs to begin with (it shouldn't), so rather than
adding further workarounds, let's get diagnostics at the source first, so we understand
what is going on.

(From OE-Core rev: 35483d4756ab53805507f72a9a0edb3f83759694)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-05 12:04:52 +00:00
Richard Purdie
a7aa37f2fd sstatesig: Add version information for find_sigingfo
Since we're changing the return values of the function, add a version so
bitbake can ensure it is using a compatible function.

(From OE-Core rev: 738bf3717002d124958a21a6925b0eca14fef131)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-05 11:59:08 +00:00
Alexander Kanavin
c45ffa9e94 sstatesig/find_siginfo: unify a disjointed API
find_siginfo() returns two different data structures depending
on whether its third argument (list of hashes to find) is empty or
not:
- a dict of timestamps keyed by path
- a dict of paths keyed by hash

This is not a good API design; it's much better to return
a dict of dicts that include both timestamp and path, keyed by
hash. Then the API consumer can decide how they want to use these
fields, particularly for additional diagnostics or informational
output.

I also took the opportunity to add a binary field that
tells if the match came from sstate or local stamps dir, which
will help prioritize local stamps when looking up most
recent task signatures.

(From OE-Core rev: 8721c52041e910bd4d8a9235b52f274f4f02c8a3)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2024-01-05 11:59:08 +00:00
Richard Purdie
52f41ff243 lib/prservice: Improve lock handling robustness
It is possible we could crash holding a lock whilst parsing in this code.
Switch to use utils.fileslocked() in the with expression to avoid this.

This may be causing some of our strange intermittent failures in PRServ
tests.

(From OE-Core rev: 4e59db15e5df2cc3d0ae042454812a2d54cef77b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-24 10:39:07 +00:00
Joshua Watt
dd5475d8e3 lib/packagedata.py: Add API to iterate over rprovides
Adds an API that makes it easier to iterate over the package data for a
all providers of a runtime dependency.

(From OE-Core rev: 68bdc219a4a819e83217f5b54c463624af8d3b9e)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-24 10:39:07 +00:00
Joshua Watt
72b6728f82 lib/packagedata.py: Fix broken symlinks for providers with a '/'
If a package had a provider with a '/' in it (e.g. "/bin/sh",
"/bin/bash", etc.), the generated symlinks were broken due to being at a
hard coded depth. Use oe.path.relsymlink() instead to make a correct
relative symbolic link

(From OE-Core rev: 8b1482a4e2adb7cf358d638265cf116b34078b84)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-23 08:46:00 +00:00
Joshua Watt
a22f7d7b53 lib/oe/path.py: Add relsymlink()
Adds API to make a relative symbolic link between two directories. The
arguments are the same as oe.path.symlink()

(From OE-Core rev: 3eeec7f3412e881e51763ef947c82772d3858f09)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-23 08:46:00 +00:00
Joshua Watt
df548e981d ipk: Switch to using zstd compression
Converts IPK package generation to use zstd instead of xz. zstd has a
much larger compression/speed tradeoff range allowing users to choose
what suits them best, and fast decompression speeds. It also continues
to support parallel compression as xz did.

A new variable called ZSTD_DEFAULTS is provided to set the defaults for
places that want to use zstd for compression; the zst image conversion
command is also modified to use this.

Finally, in order for this to function properly, opkg must include zstd
support, so it is enabled all the time with no PACKAGECONFIG to turn it
off.

(From OE-Core rev: 1bc3e9bbaa670b6128c74c76b4b5264e60ce3463)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-23 08:46:00 +00:00
Alexander Kanavin
95c187e54e sstatesig/find_siginfo: special-case gcc-source when looking in sstate caches
This is already done for local stamps just above, and will allow enabling
the full selftest that compares gcc-source signatures via printdiff
(that is, both local stamp and sstate variants).

(From OE-Core rev: 29775b5ecfc8d811293962f050fcfc3b3ad7efde)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-09 19:17:11 +00:00
Peter Kjellerstedt
3ef9ea88f1 lib/oe/recipeutils: Avoid wrapping any SRC_URI[sha*sum] variables
Before, a variable such as SRC_URI[sha512sum] would end up as:

SRC_URI[sha512sum] = "45ff3abce4dab24a8090409e6d7bb26afa7fa7812a51e067 \
28c2aa47d5b4de610d97ba4609cf13d9173087bd909fdf377235eee988a6fdcf52abb7 \
0341c40b5b"

when updated by patch_recipe_lines().

(From OE-Core rev: a67e2feed1420739504d2a59d018dff7e6e17e04)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-08 16:58:34 +00:00
Etienne Cordonnier
959b1f7de4 gdb/systemd: enable minidebuginfo support conditionally
Enabling minidebuginfo is not useful if gdb and systemd-coredump
are unable to parse it.

In order to parse it, gdb needs xz support. Systemd needs coredump enabled, as
well as elfutil enabled as well (systemd-coredump loads libdw which is part of elfutils using dlopen).

(From OE-Core rev: 0d2df803bebfd7e832ab7da54c4dacaaeeb424a9)

Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:50 +00:00
Julien Stephan
0ae9cf2373 recipeutils: bbappend_recipe: allow to patch the recipe itself
Add a new parameter update_original_recipe to allow to patch a recipe
instead of creating/updating a bbappend

(From OE-Core rev: 2f68ab2464bfad1b377df44a7b51203df59d66ce)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:49 +00:00
Julien Stephan
165626f7b9 recipeutils: bbappend_recipe: remove old srcuri entry if parameters are different
Currently we do not add a new src_ury entry if the entry already exists
AND the parameters are the same.

I believe that when an entry already exist with different parameters,
we should remove it and add the new entry otherwise we end up with two
entries with different parameters

(From OE-Core rev: a4628fffcfecb5cd95dc2558dfd39ebd71121eab)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:49 +00:00
Julien Stephan
11d4d437d5 recipeutils: bbappend_recipe: add a way to specify the name of the file to add
bbappend_recipe can take a dict of source files to add to SRC_URI where
the key is the full path to the file to be added and the value is a dict

Add a new optionnal entry "newname" to specify the name of the newly added file

(From OE-Core rev: e7bc09e5c9d7a0f4f8f4eba40730b68857b00677)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:49 +00:00
Julien Stephan
b45cab4e1c recipeutils: bbappend_recipe: fix docstring
Add missing patchdir key in docstring description for srcfiles
parameters.

Also fix typo: value --> key in srcfile docstring description

(From OE-Core rev: 302dccaef46418d0e0731870a882b54a525739ed)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:49 +00:00
Julien Stephan
eb0b664c8c recipeutils: bbappend_recipe: fix undefined variable
In the case get_bbappend_path returns None (could not find the layer
containing the recipe) the error message tries to print the recipefile,
but it is not defined. Fix it.

(From OE-Core rev: 234111fb67ffbcc5492cb0cd96db25ed8f5acea0)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:49 +00:00
Yoann Congal
2c59f5ad01 lib/oe/patch: handle creating patches for CRLF sources
Using devtool to patch CRLF based sources creates patch files which have
mixed end of lines : LF for headers and CRLF for source context and
modified lines.

Python open(..., newline=None) (default for newline arg)does detect
end-of-line in this mixed file but only outputs LF EOL data. This
result in patch files that does not apply on the original sources.

Switching to open(..., newline='') allows to detect end-of-line but keep
the original end-of-line intact. This generate correct patches for CRLF
based sources.

Fixes [YOCTO #15285]

(From OE-Core rev: 58f845499c0277a2b8069eefa235430b5f5f7661)

Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-06 22:55:49 +00:00
Julien Stephan
89f1662484 devtool: add support for git submodules
Adding the support of submodules required a lot of changes on the
internal data structures:
* initial_rev/startcommit used as a starting point for looking at new
  / updated commits was replaced by a dictionary where the keys are the
  submodule name ("." for main repo) and the values are the
  initial_rev/startcommit

* the extractPatches function now extracts patch for the main repo and
  for all submodules and stores them in a hierarchical way describing the
    submodule path

* store initial_rev/commit also for all submodules inside the recipe
  bbappend file

* _export_patches now returns dictionaries that contains the 'patchdir'
  parameter (if any). This parameter is used to add the correct
  'patchdir=' parameter on the recipe

Also, recipe can extract a secondary git tree inside the workdir.

By default, at the end of the do_patch function, there is a hook in
devtool that commits everything that was modified to have a clean
repository. It uses the command: "git add .; git commit ..."

The issue here is that, it adds the secondary git tree as a submodule
but in a wrong way. Doing "git add <git dir>" declares a submodule but do
not adds a url associated to it, and all following "git submodule foreach"
commands will fail.

So detect that a git tree was extracted inside S and correctly add it
using "git submodule add <url> <path>", so that it will be considered as a
regular git submodule

(From OE-Core rev: 900129cbdf25297a42ab5dbd02d1adbea405c935)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-12-01 11:48:25 +00:00
Julien Stephan
c7ed37b5ff lib/oe/recipeutils.py: remove trailing white-spaces
Remove useless trailing white-spaces

(From OE-Core rev: 4aa28d5ea74016087979afd1faea478fda253cc3)

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-30 08:43:04 +00:00
Ross Burton
f841d7918d lib/oe/patch: ensure os.chdir restoring always happens
If we chdir(), do the chdir back to the original directory in a finally
block so they always run.

(From OE-Core rev: cdc40292818683b6df1c814498c7589450a163fa)

Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-22 14:08:48 +00:00
Richard Purdie
0769e8ef6b bitbake.conf: Drop oldincludedir
Autoconf defines this as:

"The directory for installing C header files for non-GCC compilers."

Whilst this is something autoconf does allow changing, I find it hard
to believe it has much use in the wild now and that headers don't get
split like this in reality, it would probably only be useful on really
old unixes.. The values are the same in our configuration anyway.

Drop the value and just use includedir everywhere.

(From OE-Core rev: 506c91cbc6a604a84e37e53ccff430436369802e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-20 15:30:52 +00:00
Richard Purdie
3d35d09ff2 sstate: Ensure sstate searches update file mtime
Commands like "bitbake XXX -S printdiff" search for sstate files but don't download
them. This means that local files aren't touched as the download code would do, meaning
the sstate cleanup scripts can delete them. This can then lead to obtuse build failures.

Have the search code touch local files in the same way as the main code paths would to
avoid these files disappearing.

Move the function to a common touch() function in lib/oe instead of duplicating code.

(From OE-Core rev: a27fc0bd5706ab5b9c68a0271fcf57377a678cdf)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-08 10:56:35 +00:00
Vyacheslav Yurkov
914a921c33 lib/oe/path: Deploy files can start only with a dot
There might be only hidden files deployed. In that case we don't need a
generic wildcard present in copy command, otherwise it fails with:

Exception: subprocess.CalledProcessError: Command 'cp -afl --preserve=xattr ./.??* ./* <BUILDDIR>/tmp/deploy/images/qemux86-64' returned non-zero exit status 1.

Subprocess output:
cp: cannot stat './*': No such file or directory

(From OE-Core rev: f92c751281609ea6bd6b838307de4bc70bf26ab9)

Signed-off-by: Vyacheslav Yurkov <Vyacheslav.Yurkov@wika.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-05 11:28:35 +00:00
Jermain Horsman
a9063f845b lib/oe/buildcfg.py: Remove unused parameter
Several functions included the 'd' parameter but never used it,
additionally the value passed is always None.

(From OE-Core rev: 9e03ce0426576ebef3739dc1dfec4f7cd73ae094)

Signed-off-by: Jermain Horsman <jermain.horsman@nedap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-05 11:28:33 +00:00
Jermain Horsman
d012de568f lib/oe/buildcfg.py: Include missing import
get_layer_revisions() uses the os.path module but it is not included
in the imports.

(From OE-Core rev: 4339ef5b46c39cce402d1cc3526b39702837b839)

Signed-off-by: Jermain Horsman <jermain.horsman@nedap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-05 11:28:32 +00:00
Alexander Kanavin
a92d76e4da meta/lib/oe/copy_buildsystem.py: do not derefence symlinks
This was added (I think) for the purpose of supporting layers that refer
to items outside of the layer via relative symlinks:
https://git.yoctoproject.org/poky-contrib/commit/?id=d31d1ad4e566e42d0bbcf1f41ac25e33181fb517

I do not think copying the link target into the layer that references it is the
correct solution: rather the original target should be included
into the SDK with the same relative path.

This change is done for the sake of preserving symlinks
that are referencing things inside the layer as they are;
particularly the content of scripts/esdk-tools/.

(From OE-Core rev: 52a7bbd5c4875c5f61ea65dda38e495a2925a20d)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-05 08:44:06 +00:00