524e6b65a6
The NPM package.json documentation[1] states that git URLs may contain a commit-ish suffix to specify a specific revision. When running `npm install`, this revision will be looked for on any branch of the repository. The bitbake implementation however translates the URL stored in package.json into a git URL to be fetch by the bitbake git fetcher. The bitbake fetcher git.py, enforces the branch to be master by default. If the revision specified in the package.json is not on the master branch, the fetch will fail while the package.json is valid. To fix this, append the ";nobranch=1" suffix to the revision in the git URL to be fetched. This will make the bitbake git fetcher ignore the branch and respect the behavior of `npm install``. This can be tested with the following command: $ devtool add --npm-dev https://github.com/seapath/cockpit-cluster-dashboard.git -B version Which points to a project which has a package.json with a git URL: ```json "devDependencies": { "cockpit-repo": "git+https://github.com/cockpit-project/cockpit.git#d34cabacb8e5e1e028c7eea3d6e3b606d862b8ac" } ``` In this repo, the specified revision is on the "main" branch, which would fail without this fix. [1] https://docs.npmjs.com/cli/v10/configuring-npm/package-json#git-urls-as-dependencies Co-authored-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com> (Bitbake rev: 37a35adf7882f231c13643dbf9168497c6a242a1) Signed-off-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com> Signed-off-by: Enguerrand de Ribaucourt <enguerrand.de-ribaucourt@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are expectations of users of the fetcher code. This file attempts to document some of the constraints that are present. Some are obvious, some are less so. It is documented in the context of how OE uses it but the API calls are generic.
a) network access for sources is only expected to happen in the do_fetch task. This is not enforced or tested but is required so that we can:
i) audit the sources used (i.e. for license/manifest reasons) ii) support offline builds with a suitable cache iii) allow work to continue even with downtime upstream iv) allow for changes upstream in incompatible ways v) allow rebuilding of the software in X years time
b) network access is not expected in do_unpack task.
c) you can take DL_DIR and use it as a mirror for offline builds.
d) access to the network is only made when explicitly configured in recipes (e.g. use of AUTOREV, or use of git tags which change revision).
e) fetcher output is deterministic (i.e. if you fetch configuration XXX now it will match in future exactly in a clean build with a new DL_DIR). One specific pain point example are git tags. They can be replaced and change so the git fetcher has to resolve them with the network. We use git revisions where possible to avoid this and ensure determinism.
f) network access is expected to work with the standard linux proxy variables so that access behind firewalls works (the fetcher sets these in the environment but only in the do_fetch tasks).
g) access during parsing has to be minimal, a "git ls-remote" for an AUTOREV git recipe might be ok but you can't expect to checkout a git tree.
h) we need to provide revision information during parsing such that a version for the recipe can be constructed.
i) versions are expected to be able to increase in a way which sorts allowing package feeds to operate (see PR server required for git revisions to sort).
j) API to query for possible version upgrades of a url is highly desireable to allow our automated upgrage code to function (it is implied this does always have network access).
k) Where fixes or changes to behaviour in the fetcher are made, we ask that test cases are added (run with "bitbake-selftest bb.tests.fetch"). We do have fairly extensive test coverage of the fetcher as it is the only way to track all of its corner cases, it still doesn't give entire coverage though sadly.
l) If using tools during parse time, they will have to be in ASSUME_PROVIDED in OE's context as we can't build git-native, then parse a recipe and use git ls-remote.
Not all fetchers support all features, autorev is optional and doesn't make sense for some. Upgrade detection means different things in different contexts too.