build-appliance-image: Update to rocko head revision

(From OE-Core rev: 9d1129076658b4c5827c95ad8b195a7a100d7a9e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

.gitignore

@@ -1,14 +1,12 @@
 *.pyc
 *.pyo
 /*.patch
-/.repo/
 /build*/
 pyshtables.py
 pstage/
 scripts/oe-git-proxy-socks
 sources/
 meta-*/
-buildtools/
 !meta-skeleton
 !meta-selftest
 hob-image-*.bb
@@ -20,13 +18,9 @@ hob-image-*.bb
 !meta-yocto
 !meta-yocto-bsp
 !meta-yocto-imported
-/documentation/*/eclipse/
-/documentation/*/*.html
-/documentation/*/*.pdf
-/documentation/*/*.tgz
-/bitbake/doc/bitbake-user-manual/bitbake-user-manual.html
-/bitbake/doc/bitbake-user-manual/bitbake-user-manual.pdf
-/bitbake/doc/bitbake-user-manual/bitbake-user-manual.tgz
+documentation/user-manual/user-manual.html
+documentation/user-manual/user-manual.pdf
+documentation/user-manual/user-manual.tgz
 pull-*/
 bitbake/lib/toaster/contrib/tts/backlog.txt
 bitbake/lib/toaster/contrib/tts/log/*

LICENSE

@@ -1,20 +1,14 @@
 Different components of OpenEmbedded are under different licenses (a mix
-of MIT and GPLv2). See LICENSE.GPL-2.0-only and LICENSE.MIT for further 
-details of the individual licenses.
+of MIT and GPLv2). Please see:
+
+meta/COPYING.GPLv2 (GPLv2)
+meta/COPYING.MIT (MIT)
+meta-selftest/COPYING.MIT (MIT)
+meta-skeleton/COPYING.MIT (MIT)
 
 All metadata is MIT licensed unless otherwise stated. Source code
-included in tree for individual recipes (e.g. patches) are under 
-the LICENSE stated in the associated recipe (.bb file) unless 
-otherwise stated.
+included in tree for individual recipes is under the LICENSE stated in
+the associated recipe (.bb file) unless otherwise stated.
 
 License information for any other files is either explicitly stated 
-or defaults to GPL version 2 only.
-
-Individual files contain the following style tags instead of the full license 
-text to identify their license:
-
-    SPDX-License-Identifier: GPL-2.0-only
-    SPDX-License-Identifier: MIT
-
-This enables machine processing of license information based on the SPDX
-License Identifiers that are here available: http://spdx.org/licenses/
+or defaults to GPL version 2.

LICENSE.GPL-2.0-only

@@ -1,288 +0,0 @@
-		    GNU GENERAL PUBLIC LICENSE
-		       Version 2, June 1991
-
- Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
- 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
- Everyone is permitted to copy and distribute verbatim copies
- of this license document, but changing it is not allowed.
-
-			    Preamble
-
-  The licenses for most software are designed to take away your
-freedom to share and change it.  By contrast, the GNU General Public
-License is intended to guarantee your freedom to share and change free
-software--to make sure the software is free for all its users.  This
-General Public License applies to most of the Free Software
-Foundation's software and to any other program whose authors commit to
-using it.  (Some other Free Software Foundation software is covered by
-the GNU Lesser General Public License instead.)  You can apply it to
-your programs, too.
-
-  When we speak of free software, we are referring to freedom, not
-price.  Our General Public Licenses are designed to make sure that you
-have the freedom to distribute copies of free software (and charge for
-this service if you wish), that you receive source code or can get it
-if you want it, that you can change the software or use pieces of it
-in new free programs; and that you know you can do these things.
-
-  To protect your rights, we need to make restrictions that forbid
-anyone to deny you these rights or to ask you to surrender the rights.
-These restrictions translate to certain responsibilities for you if you
-distribute copies of the software, or if you modify it.
-
-  For example, if you distribute copies of such a program, whether
-gratis or for a fee, you must give the recipients all the rights that
-you have.  You must make sure that they, too, receive or can get the
-source code.  And you must show them these terms so they know their
-rights.
-
-  We protect your rights with two steps: (1) copyright the software, and
-(2) offer you this license which gives you legal permission to copy,
-distribute and/or modify the software.
-
-  Also, for each author's protection and ours, we want to make certain
-that everyone understands that there is no warranty for this free
-software.  If the software is modified by someone else and passed on, we
-want its recipients to know that what they have is not the original, so
-that any problems introduced by others will not reflect on the original
-authors' reputations.
-
-  Finally, any free program is threatened constantly by software
-patents.  We wish to avoid the danger that redistributors of a free
-program will individually obtain patent licenses, in effect making the
-program proprietary.  To prevent this, we have made it clear that any
-patent must be licensed for everyone's free use or not licensed at all.
-
-  The precise terms and conditions for copying, distribution and
-modification follow.
-
-		    GNU GENERAL PUBLIC LICENSE
-   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
-
-  0. This License applies to any program or other work which contains
-a notice placed by the copyright holder saying it may be distributed
-under the terms of this General Public License.  The "Program", below,
-refers to any such program or work, and a "work based on the Program"
-means either the Program or any derivative work under copyright law:
-that is to say, a work containing the Program or a portion of it,
-either verbatim or with modifications and/or translated into another
-language.  (Hereinafter, translation is included without limitation in
-the term "modification".)  Each licensee is addressed as "you".
-
-Activities other than copying, distribution and modification are not
-covered by this License; they are outside its scope.  The act of
-running the Program is not restricted, and the output from the Program
-is covered only if its contents constitute a work based on the
-Program (independent of having been made by running the Program).
-Whether that is true depends on what the Program does.
-
-  1. You may copy and distribute verbatim copies of the Program's
-source code as you receive it, in any medium, provided that you
-conspicuously and appropriately publish on each copy an appropriate
-copyright notice and disclaimer of warranty; keep intact all the
-notices that refer to this License and to the absence of any warranty;
-and give any other recipients of the Program a copy of this License
-along with the Program.
-
-You may charge a fee for the physical act of transferring a copy, and
-you may at your option offer warranty protection in exchange for a fee.
-
-  2. You may modify your copy or copies of the Program or any portion
-of it, thus forming a work based on the Program, and copy and
-distribute such modifications or work under the terms of Section 1
-above, provided that you also meet all of these conditions:
-
-    a) You must cause the modified files to carry prominent notices
-    stating that you changed the files and the date of any change.
-
-    b) You must cause any work that you distribute or publish, that in
-    whole or in part contains or is derived from the Program or any
-    part thereof, to be licensed as a whole at no charge to all third
-    parties under the terms of this License.
-
-    c) If the modified program normally reads commands interactively
-    when run, you must cause it, when started running for such
-    interactive use in the most ordinary way, to print or display an
-    announcement including an appropriate copyright notice and a
-    notice that there is no warranty (or else, saying that you provide
-    a warranty) and that users may redistribute the program under
-    these conditions, and telling the user how to view a copy of this
-    License.  (Exception: if the Program itself is interactive but
-    does not normally print such an announcement, your work based on
-    the Program is not required to print an announcement.)
-
-These requirements apply to the modified work as a whole.  If
-identifiable sections of that work are not derived from the Program,
-and can be reasonably considered independent and separate works in
-themselves, then this License, and its terms, do not apply to those
-sections when you distribute them as separate works.  But when you
-distribute the same sections as part of a whole which is a work based
-on the Program, the distribution of the whole must be on the terms of
-this License, whose permissions for other licensees extend to the
-entire whole, and thus to each and every part regardless of who wrote it.
-
-Thus, it is not the intent of this section to claim rights or contest
-your rights to work written entirely by you; rather, the intent is to
-exercise the right to control the distribution of derivative or
-collective works based on the Program.
-
-In addition, mere aggregation of another work not based on the Program
-with the Program (or with a work based on the Program) on a volume of
-a storage or distribution medium does not bring the other work under
-the scope of this License.
-
-  3. You may copy and distribute the Program (or a work based on it,
-under Section 2) in object code or executable form under the terms of
-Sections 1 and 2 above provided that you also do one of the following:
-
-    a) Accompany it with the complete corresponding machine-readable
-    source code, which must be distributed under the terms of Sections
-    1 and 2 above on a medium customarily used for software interchange; or,
-
-    b) Accompany it with a written offer, valid for at least three
-    years, to give any third party, for a charge no more than your
-    cost of physically performing source distribution, a complete
-    machine-readable copy of the corresponding source code, to be
-    distributed under the terms of Sections 1 and 2 above on a medium
-    customarily used for software interchange; or,
-
-    c) Accompany it with the information you received as to the offer
-    to distribute corresponding source code.  (This alternative is
-    allowed only for noncommercial distribution and only if you
-    received the program in object code or executable form with such
-    an offer, in accord with Subsection b above.)
-
-The source code for a work means the preferred form of the work for
-making modifications to it.  For an executable work, complete source
-code means all the source code for all modules it contains, plus any
-associated interface definition files, plus the scripts used to
-control compilation and installation of the executable.  However, as a
-special exception, the source code distributed need not include
-anything that is normally distributed (in either source or binary
-form) with the major components (compiler, kernel, and so on) of the
-operating system on which the executable runs, unless that component
-itself accompanies the executable.
-
-If distribution of executable or object code is made by offering
-access to copy from a designated place, then offering equivalent
-access to copy the source code from the same place counts as
-distribution of the source code, even though third parties are not
-compelled to copy the source along with the object code.
-
-  4. You may not copy, modify, sublicense, or distribute the Program
-except as expressly provided under this License.  Any attempt
-otherwise to copy, modify, sublicense or distribute the Program is
-void, and will automatically terminate your rights under this License.
-However, parties who have received copies, or rights, from you under
-this License will not have their licenses terminated so long as such
-parties remain in full compliance.
-
-  5. You are not required to accept this License, since you have not
-signed it.  However, nothing else grants you permission to modify or
-distribute the Program or its derivative works.  These actions are
-prohibited by law if you do not accept this License.  Therefore, by
-modifying or distributing the Program (or any work based on the
-Program), you indicate your acceptance of this License to do so, and
-all its terms and conditions for copying, distributing or modifying
-the Program or works based on it.
-
-  6. Each time you redistribute the Program (or any work based on the
-Program), the recipient automatically receives a license from the
-original licensor to copy, distribute or modify the Program subject to
-these terms and conditions.  You may not impose any further
-restrictions on the recipients' exercise of the rights granted herein.
-You are not responsible for enforcing compliance by third parties to
-this License.
-
-  7. If, as a consequence of a court judgment or allegation of patent
-infringement or for any other reason (not limited to patent issues),
-conditions are imposed on you (whether by court order, agreement or
-otherwise) that contradict the conditions of this License, they do not
-excuse you from the conditions of this License.  If you cannot
-distribute so as to satisfy simultaneously your obligations under this
-License and any other pertinent obligations, then as a consequence you
-may not distribute the Program at all.  For example, if a patent
-license would not permit royalty-free redistribution of the Program by
-all those who receive copies directly or indirectly through you, then
-the only way you could satisfy both it and this License would be to
-refrain entirely from distribution of the Program.
-
-If any portion of this section is held invalid or unenforceable under
-any particular circumstance, the balance of the section is intended to
-apply and the section as a whole is intended to apply in other
-circumstances.
-
-It is not the purpose of this section to induce you to infringe any
-patents or other property right claims or to contest validity of any
-such claims; this section has the sole purpose of protecting the
-integrity of the free software distribution system, which is
-implemented by public license practices.  Many people have made
-generous contributions to the wide range of software distributed
-through that system in reliance on consistent application of that
-system; it is up to the author/donor to decide if he or she is willing
-to distribute software through any other system and a licensee cannot
-impose that choice.
-
-This section is intended to make thoroughly clear what is believed to
-be a consequence of the rest of this License.
-
-  8. If the distribution and/or use of the Program is restricted in
-certain countries either by patents or by copyrighted interfaces, the
-original copyright holder who places the Program under this License
-may add an explicit geographical distribution limitation excluding
-those countries, so that distribution is permitted only in or among
-countries not thus excluded.  In such case, this License incorporates
-the limitation as if written in the body of this License.
-
-  9. The Free Software Foundation may publish revised and/or new versions
-of the General Public License from time to time.  Such new versions will
-be similar in spirit to the present version, but may differ in detail to
-address new problems or concerns.
-
-Each version is given a distinguishing version number.  If the Program
-specifies a version number of this License which applies to it and "any
-later version", you have the option of following the terms and conditions
-either of that version or of any later version published by the Free
-Software Foundation.  If the Program does not specify a version number of
-this License, you may choose any version ever published by the Free Software
-Foundation.
-
-  10. If you wish to incorporate parts of the Program into other free
-programs whose distribution conditions are different, write to the author
-to ask for permission.  For software which is copyrighted by the Free
-Software Foundation, write to the Free Software Foundation; we sometimes
-make exceptions for this.  Our decision will be guided by the two goals
-of preserving the free status of all derivatives of our free software and
-of promoting the sharing and reuse of software generally.
-
-			    NO WARRANTY
-
-  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
-FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
-OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
-PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
-OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
-MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
-TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
-PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
-REPAIR OR CORRECTION.
-
-  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
-WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
-REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
-INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
-OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
-TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
-YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
-PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
-POSSIBILITY OF SUCH DAMAGES.
-
-		     END OF TERMS AND CONDITIONS
-
-Note:
-Individual files contain the following tag instead of the full license text.
-
-    SPDX-License-Identifier: GPL-2.0-only
-
-This enables machine processing of license information based on the SPDX
-License Identifiers that are here available: http://spdx.org/licenses/

LICENSE.MIT

@@ -1,25 +0,0 @@
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in
-all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
-
-Note:
-Individual files contain the following tag instead of the full license text.
-
-    SPDX-License-Identifier: MIT
-
-This enables machine processing of license information based on the SPDX
-License Identifiers that are here available: http://spdx.org/licenses/

MEMORIAM

@@ -1,5 +0,0 @@
-Some project contributors who are sadly no longer with us:
-
-Greg Gilbert (treke) - Ahead of his time with licensing
-Thomas Wood (thos) - Creator of the original sato
-Scott Rifenbark (scottrif) - Our long standing techwriter whose words live on

README.LSB

@@ -0,0 +1,25 @@
+OE-Core aims to be able to provide basic LSB compatible images. There
+are some challenges for OE as LSB isn't always 100% relevant to its
+target embedded and IoT audiences. 
+
+One challenge is that the LSB spec is no longer being actively
+developed [https://github.com/LinuxStandardBase/lsb] and has 
+components which are end of life or significantly dated. OE 
+therefore provides compatibility with the following caveats:
+
+* Qt4 is provided by the separate meta-qt4 layer. Its noted that Qt4 
+  is end of life and this isn't something the core project regularly 
+  tests any longer. Users are recommended to group together to support
+  maintenance of that layer. [http://git.yoctoproject.org/cgit/cgit.cgi/meta-qt4/]
+
+* mailx has been dropped since its no longer being developed upstream 
+  and there are better, more modern replacements such as s-nail 
+  (http://sdaoden.eu/code.html) or mailutils (http://mailutils.org/).
+
+* A few perl modules that were required by LSB 4.x aren't provided:
+  libclass-isa, libenv, libdumpvalue, libfile-checktree,
+  libi18n-collate, libpod-plainer.
+
+* libpng 1.2 isn't provided; oe-core includes the latest release of libpng
+  instead.
+

README.OE-Core

@@ -1,29 +0,0 @@
-OpenEmbedded-Core
-=================
-
-OpenEmbedded-Core is a layer containing the core metadata for current versions
-of OpenEmbedded. It is distro-less (can build a functional image with
-DISTRO = "nodistro") and contains only emulated machine support.
-
-For information about OpenEmbedded, see the OpenEmbedded website:
-    https://www.openembedded.org/
-
-The Yocto Project has extensive documentation about OE including a reference manual
-which can be found at:
-    https://docs.yoctoproject.org/
-
-
-Contributing
-------------
-
-Please refer to
-https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded
-for guidelines on how to submit patches.
-
-Mailing list:
-
-    https://lists.openembedded.org/g/openembedded-core
-
-Source code:
-
-    https://git.openembedded.org/openembedded-core/

SECURITY.md

@@ -1,24 +0,0 @@
-How to Report a Potential Vulnerability?
-========================================
-
-If you would like to report a public issue (for example, one with a released
-CVE number), please report it using the
-[https://bugzilla.yoctoproject.org/enter_bug.cgi?product=Security Security Bugzilla].
-If you have a patch ready, submit it following the same procedure as any other
-patch as described in README.md.
-
-If you are dealing with a not-yet released or urgent issue, please send a
-message to security AT yoctoproject DOT org, including as many details as
-possible: the layer or software module affected, the recipe and its version,
-and any example code, if available.
-
-Branches maintained with security fixes
----------------------------------------
-
-See [https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS Stable release and LTS]
-for detailed info regarding the policies and maintenance of Stable branches.
-
-The [https://wiki.yoctoproject.org/wiki/Releases Release page] contains a list of all
-releases of the Yocto Project. Versions in grey are no longer actively maintained with
-security patches, but well-tested patches may still be accepted for them for
-significant issues.

bitbake/.gitattributes

@@ -1,2 +0,0 @@
-*min.js binary
-*min.css binary

bitbake/COPYING

@@ -0,0 +1,339 @@
+		    GNU GENERAL PUBLIC LICENSE
+		       Version 2, June 1991
+
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
+ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+			    Preamble
+
+  The licenses for most software are designed to take away your
+freedom to share and change it.  By contrast, the GNU General Public
+License is intended to guarantee your freedom to share and change free
+software--to make sure the software is free for all its users.  This
+General Public License applies to most of the Free Software
+Foundation's software and to any other program whose authors commit to
+using it.  (Some other Free Software Foundation software is covered by
+the GNU Lesser General Public License instead.)  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get it
+if you want it, that you can change the software or use pieces of it
+in new free programs; and that you know you can do these things.
+
+  To protect your rights, we need to make restrictions that forbid
+anyone to deny you these rights or to ask you to surrender the rights.
+These restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must give the recipients all the rights that
+you have.  You must make sure that they, too, receive or can get the
+source code.  And you must show them these terms so they know their
+rights.
+
+  We protect your rights with two steps: (1) copyright the software, and
+(2) offer you this license which gives you legal permission to copy,
+distribute and/or modify the software.
+
+  Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software.  If the software is modified by someone else and passed on, we
+want its recipients to know that what they have is not the original, so
+that any problems introduced by others will not reflect on the original
+authors' reputations.
+
+  Finally, any free program is threatened constantly by software
+patents.  We wish to avoid the danger that redistributors of a free
+program will individually obtain patent licenses, in effect making the
+program proprietary.  To prevent this, we have made it clear that any
+patent must be licensed for everyone's free use or not licensed at all.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+		    GNU GENERAL PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. This License applies to any program or other work which contains
+a notice placed by the copyright holder saying it may be distributed
+under the terms of this General Public License.  The "Program", below,
+refers to any such program or work, and a "work based on the Program"
+means either the Program or any derivative work under copyright law:
+that is to say, a work containing the Program or a portion of it,
+either verbatim or with modifications and/or translated into another
+language.  (Hereinafter, translation is included without limitation in
+the term "modification".)  Each licensee is addressed as "you".
+
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope.  The act of
+running the Program is not restricted, and the output from the Program
+is covered only if its contents constitute a work based on the
+Program (independent of having been made by running the Program).
+Whether that is true depends on what the Program does.
+
+  1. You may copy and distribute verbatim copies of the Program's
+source code as you receive it, in any medium, provided that you
+conspicuously and appropriately publish on each copy an appropriate
+copyright notice and disclaimer of warranty; keep intact all the
+notices that refer to this License and to the absence of any warranty;
+and give any other recipients of the Program a copy of this License
+along with the Program.
+
+You may charge a fee for the physical act of transferring a copy, and
+you may at your option offer warranty protection in exchange for a fee.
+
+  2. You may modify your copy or copies of the Program or any portion
+of it, thus forming a work based on the Program, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+    a) You must cause the modified files to carry prominent notices
+    stating that you changed the files and the date of any change.
+
+    b) You must cause any work that you distribute or publish, that in
+    whole or in part contains or is derived from the Program or any
+    part thereof, to be licensed as a whole at no charge to all third
+    parties under the terms of this License.
+
+    c) If the modified program normally reads commands interactively
+    when run, you must cause it, when started running for such
+    interactive use in the most ordinary way, to print or display an
+    announcement including an appropriate copyright notice and a
+    notice that there is no warranty (or else, saying that you provide
+    a warranty) and that users may redistribute the program under
+    these conditions, and telling the user how to view a copy of this
+    License.  (Exception: if the Program itself is interactive but
+    does not normally print such an announcement, your work based on
+    the Program is not required to print an announcement.)
+
+These requirements apply to the modified work as a whole.  If
+identifiable sections of that work are not derived from the Program,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works.  But when you
+distribute the same sections as part of a whole which is a work based
+on the Program, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+  3. You may copy and distribute the Program (or a work based on it,
+under Section 2) in object code or executable form under the terms of
+Sections 1 and 2 above provided that you also do one of the following:
+
+    a) Accompany it with the complete corresponding machine-readable
+    source code, which must be distributed under the terms of Sections
+    1 and 2 above on a medium customarily used for software interchange; or,
+
+    b) Accompany it with a written offer, valid for at least three
+    years, to give any third party, for a charge no more than your
+    cost of physically performing source distribution, a complete
+    machine-readable copy of the corresponding source code, to be
+    distributed under the terms of Sections 1 and 2 above on a medium
+    customarily used for software interchange; or,
+
+    c) Accompany it with the information you received as to the offer
+    to distribute corresponding source code.  (This alternative is
+    allowed only for noncommercial distribution and only if you
+    received the program in object code or executable form with such
+    an offer, in accord with Subsection b above.)
+
+The source code for a work means the preferred form of the work for
+making modifications to it.  For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to
+control compilation and installation of the executable.  However, as a
+special exception, the source code distributed need not include
+anything that is normally distributed (in either source or binary
+form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component
+itself accompanies the executable.
+
+If distribution of executable or object code is made by offering
+access to copy from a designated place, then offering equivalent
+access to copy the source code from the same place counts as
+distribution of the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+  4. You may not copy, modify, sublicense, or distribute the Program
+except as expressly provided under this License.  Any attempt
+otherwise to copy, modify, sublicense or distribute the Program is
+void, and will automatically terminate your rights under this License.
+However, parties who have received copies, or rights, from you under
+this License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+  5. You are not required to accept this License, since you have not
+signed it.  However, nothing else grants you permission to modify or
+distribute the Program or its derivative works.  These actions are
+prohibited by law if you do not accept this License.  Therefore, by
+modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Program or works based on it.
+
+  6. Each time you redistribute the Program (or any work based on the
+Program), the recipient automatically receives a license from the
+original licensor to copy, distribute or modify the Program subject to
+these terms and conditions.  You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties to
+this License.
+
+  7. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Program at all.  For example, if a patent
+license would not permit royalty-free redistribution of the Program by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system, which is
+implemented by public license practices.  Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+  8. If the distribution and/or use of the Program is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Program under this License
+may add an explicit geographical distribution limitation excluding
+those countries, so that distribution is permitted only in or among
+countries not thus excluded.  In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+  9. The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+Each version is given a distinguishing version number.  If the Program
+specifies a version number of this License which applies to it and "any
+later version", you have the option of following the terms and conditions
+either of that version or of any later version published by the Free
+Software Foundation.  If the Program does not specify a version number of
+this License, you may choose any version ever published by the Free Software
+Foundation.
+
+  10. If you wish to incorporate parts of the Program into other free
+programs whose distribution conditions are different, write to the author
+to ask for permission.  For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this.  Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software and
+of promoting the sharing and reuse of software generally.
+
+			    NO WARRANTY
+
+  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+		     END OF TERMS AND CONDITIONS
+
+	    How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+Also add information on how to contact you by electronic and paper mail.
+
+If the program is interactive, make it output a short notice like this
+when it starts in an interactive mode:
+
+    Gnomovision version 69, Copyright (C) year name of author
+    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, the commands you use may
+be called something other than `show w' and `show c'; they could even be
+mouse-clicks or menu items--whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary.  Here is a sample; alter the names:
+
+  Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+  `Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+  <signature of Ty Coon>, 1 April 1989
+  Ty Coon, President of Vice
+
+This General Public License does not permit incorporating your program into
+proprietary programs.  If your program is a subroutine library, you may
+consider it more useful to permit linking proprietary applications with the
+library.  If this is what you want to do, use the GNU Lesser General
+Public License instead of this License.

bitbake/HEADER

@@ -0,0 +1,19 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# <one line to give the program's name and a brief idea of what it does.>
+# Copyright (C) <year>  <name of author>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+

bitbake/LICENSE

@@ -1,13 +1,4 @@
-BitBake is licensed under the GNU General Public License version 2.0. See 
-LICENSE.GPL-2.0-only for further details.
-
-Individual files contain the following style tags instead of the full license text:
-
-    SPDX-License-Identifier:	GPL-2.0-only
-
-This enables machine processing of license information based on the SPDX
-License Identifiers that are here available: http://spdx.org/licenses/
-
+BitBake is licensed under the GNU General Public License version 2.0. See COPYING for further details.
 
 The following external components are distributed with this software:
 
@@ -26,4 +17,3 @@ Foundation and individual contributors.
 * Font Awesome fonts redistributed under the SIL Open Font License 1.1
 
 * simplediff is distributed under the zlib license.
-

bitbake/LICENSE.GPL-2.0-only

@@ -1,288 +0,0 @@
-		    GNU GENERAL PUBLIC LICENSE
-		       Version 2, June 1991
-
- Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
- 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
- Everyone is permitted to copy and distribute verbatim copies
- of this license document, but changing it is not allowed.
-
-			    Preamble
-
-  The licenses for most software are designed to take away your
-freedom to share and change it.  By contrast, the GNU General Public
-License is intended to guarantee your freedom to share and change free
-software--to make sure the software is free for all its users.  This
-General Public License applies to most of the Free Software
-Foundation's software and to any other program whose authors commit to
-using it.  (Some other Free Software Foundation software is covered by
-the GNU Lesser General Public License instead.)  You can apply it to
-your programs, too.
-
-  When we speak of free software, we are referring to freedom, not
-price.  Our General Public Licenses are designed to make sure that you
-have the freedom to distribute copies of free software (and charge for
-this service if you wish), that you receive source code or can get it
-if you want it, that you can change the software or use pieces of it
-in new free programs; and that you know you can do these things.
-
-  To protect your rights, we need to make restrictions that forbid
-anyone to deny you these rights or to ask you to surrender the rights.
-These restrictions translate to certain responsibilities for you if you
-distribute copies of the software, or if you modify it.
-
-  For example, if you distribute copies of such a program, whether
-gratis or for a fee, you must give the recipients all the rights that
-you have.  You must make sure that they, too, receive or can get the
-source code.  And you must show them these terms so they know their
-rights.
-
-  We protect your rights with two steps: (1) copyright the software, and
-(2) offer you this license which gives you legal permission to copy,
-distribute and/or modify the software.
-
-  Also, for each author's protection and ours, we want to make certain
-that everyone understands that there is no warranty for this free
-software.  If the software is modified by someone else and passed on, we
-want its recipients to know that what they have is not the original, so
-that any problems introduced by others will not reflect on the original
-authors' reputations.
-
-  Finally, any free program is threatened constantly by software
-patents.  We wish to avoid the danger that redistributors of a free
-program will individually obtain patent licenses, in effect making the
-program proprietary.  To prevent this, we have made it clear that any
-patent must be licensed for everyone's free use or not licensed at all.
-
-  The precise terms and conditions for copying, distribution and
-modification follow.
-
-		    GNU GENERAL PUBLIC LICENSE
-   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
-
-  0. This License applies to any program or other work which contains
-a notice placed by the copyright holder saying it may be distributed
-under the terms of this General Public License.  The "Program", below,
-refers to any such program or work, and a "work based on the Program"
-means either the Program or any derivative work under copyright law:
-that is to say, a work containing the Program or a portion of it,
-either verbatim or with modifications and/or translated into another
-language.  (Hereinafter, translation is included without limitation in
-the term "modification".)  Each licensee is addressed as "you".
-
-Activities other than copying, distribution and modification are not
-covered by this License; they are outside its scope.  The act of
-running the Program is not restricted, and the output from the Program
-is covered only if its contents constitute a work based on the
-Program (independent of having been made by running the Program).
-Whether that is true depends on what the Program does.
-
-  1. You may copy and distribute verbatim copies of the Program's
-source code as you receive it, in any medium, provided that you
-conspicuously and appropriately publish on each copy an appropriate
-copyright notice and disclaimer of warranty; keep intact all the
-notices that refer to this License and to the absence of any warranty;
-and give any other recipients of the Program a copy of this License
-along with the Program.
-
-You may charge a fee for the physical act of transferring a copy, and
-you may at your option offer warranty protection in exchange for a fee.
-
-  2. You may modify your copy or copies of the Program or any portion
-of it, thus forming a work based on the Program, and copy and
-distribute such modifications or work under the terms of Section 1
-above, provided that you also meet all of these conditions:
-
-    a) You must cause the modified files to carry prominent notices
-    stating that you changed the files and the date of any change.
-
-    b) You must cause any work that you distribute or publish, that in
-    whole or in part contains or is derived from the Program or any
-    part thereof, to be licensed as a whole at no charge to all third
-    parties under the terms of this License.
-
-    c) If the modified program normally reads commands interactively
-    when run, you must cause it, when started running for such
-    interactive use in the most ordinary way, to print or display an
-    announcement including an appropriate copyright notice and a
-    notice that there is no warranty (or else, saying that you provide
-    a warranty) and that users may redistribute the program under
-    these conditions, and telling the user how to view a copy of this
-    License.  (Exception: if the Program itself is interactive but
-    does not normally print such an announcement, your work based on
-    the Program is not required to print an announcement.)
-
-These requirements apply to the modified work as a whole.  If
-identifiable sections of that work are not derived from the Program,
-and can be reasonably considered independent and separate works in
-themselves, then this License, and its terms, do not apply to those
-sections when you distribute them as separate works.  But when you
-distribute the same sections as part of a whole which is a work based
-on the Program, the distribution of the whole must be on the terms of
-this License, whose permissions for other licensees extend to the
-entire whole, and thus to each and every part regardless of who wrote it.
-
-Thus, it is not the intent of this section to claim rights or contest
-your rights to work written entirely by you; rather, the intent is to
-exercise the right to control the distribution of derivative or
-collective works based on the Program.
-
-In addition, mere aggregation of another work not based on the Program
-with the Program (or with a work based on the Program) on a volume of
-a storage or distribution medium does not bring the other work under
-the scope of this License.
-
-  3. You may copy and distribute the Program (or a work based on it,
-under Section 2) in object code or executable form under the terms of
-Sections 1 and 2 above provided that you also do one of the following:
-
-    a) Accompany it with the complete corresponding machine-readable
-    source code, which must be distributed under the terms of Sections
-    1 and 2 above on a medium customarily used for software interchange; or,
-
-    b) Accompany it with a written offer, valid for at least three
-    years, to give any third party, for a charge no more than your
-    cost of physically performing source distribution, a complete
-    machine-readable copy of the corresponding source code, to be
-    distributed under the terms of Sections 1 and 2 above on a medium
-    customarily used for software interchange; or,
-
-    c) Accompany it with the information you received as to the offer
-    to distribute corresponding source code.  (This alternative is
-    allowed only for noncommercial distribution and only if you
-    received the program in object code or executable form with such
-    an offer, in accord with Subsection b above.)
-
-The source code for a work means the preferred form of the work for
-making modifications to it.  For an executable work, complete source
-code means all the source code for all modules it contains, plus any
-associated interface definition files, plus the scripts used to
-control compilation and installation of the executable.  However, as a
-special exception, the source code distributed need not include
-anything that is normally distributed (in either source or binary
-form) with the major components (compiler, kernel, and so on) of the
-operating system on which the executable runs, unless that component
-itself accompanies the executable.
-
-If distribution of executable or object code is made by offering
-access to copy from a designated place, then offering equivalent
-access to copy the source code from the same place counts as
-distribution of the source code, even though third parties are not
-compelled to copy the source along with the object code.
-
-  4. You may not copy, modify, sublicense, or distribute the Program
-except as expressly provided under this License.  Any attempt
-otherwise to copy, modify, sublicense or distribute the Program is
-void, and will automatically terminate your rights under this License.
-However, parties who have received copies, or rights, from you under
-this License will not have their licenses terminated so long as such
-parties remain in full compliance.
-
-  5. You are not required to accept this License, since you have not
-signed it.  However, nothing else grants you permission to modify or
-distribute the Program or its derivative works.  These actions are
-prohibited by law if you do not accept this License.  Therefore, by
-modifying or distributing the Program (or any work based on the
-Program), you indicate your acceptance of this License to do so, and
-all its terms and conditions for copying, distributing or modifying
-the Program or works based on it.
-
-  6. Each time you redistribute the Program (or any work based on the
-Program), the recipient automatically receives a license from the
-original licensor to copy, distribute or modify the Program subject to
-these terms and conditions.  You may not impose any further
-restrictions on the recipients' exercise of the rights granted herein.
-You are not responsible for enforcing compliance by third parties to
-this License.
-
-  7. If, as a consequence of a court judgment or allegation of patent
-infringement or for any other reason (not limited to patent issues),
-conditions are imposed on you (whether by court order, agreement or
-otherwise) that contradict the conditions of this License, they do not
-excuse you from the conditions of this License.  If you cannot
-distribute so as to satisfy simultaneously your obligations under this
-License and any other pertinent obligations, then as a consequence you
-may not distribute the Program at all.  For example, if a patent
-license would not permit royalty-free redistribution of the Program by
-all those who receive copies directly or indirectly through you, then
-the only way you could satisfy both it and this License would be to
-refrain entirely from distribution of the Program.
-
-If any portion of this section is held invalid or unenforceable under
-any particular circumstance, the balance of the section is intended to
-apply and the section as a whole is intended to apply in other
-circumstances.
-
-It is not the purpose of this section to induce you to infringe any
-patents or other property right claims or to contest validity of any
-such claims; this section has the sole purpose of protecting the
-integrity of the free software distribution system, which is
-implemented by public license practices.  Many people have made
-generous contributions to the wide range of software distributed
-through that system in reliance on consistent application of that
-system; it is up to the author/donor to decide if he or she is willing
-to distribute software through any other system and a licensee cannot
-impose that choice.
-
-This section is intended to make thoroughly clear what is believed to
-be a consequence of the rest of this License.
-
-  8. If the distribution and/or use of the Program is restricted in
-certain countries either by patents or by copyrighted interfaces, the
-original copyright holder who places the Program under this License
-may add an explicit geographical distribution limitation excluding
-those countries, so that distribution is permitted only in or among
-countries not thus excluded.  In such case, this License incorporates
-the limitation as if written in the body of this License.
-
-  9. The Free Software Foundation may publish revised and/or new versions
-of the General Public License from time to time.  Such new versions will
-be similar in spirit to the present version, but may differ in detail to
-address new problems or concerns.
-
-Each version is given a distinguishing version number.  If the Program
-specifies a version number of this License which applies to it and "any
-later version", you have the option of following the terms and conditions
-either of that version or of any later version published by the Free
-Software Foundation.  If the Program does not specify a version number of
-this License, you may choose any version ever published by the Free Software
-Foundation.
-
-  10. If you wish to incorporate parts of the Program into other free
-programs whose distribution conditions are different, write to the author
-to ask for permission.  For software which is copyrighted by the Free
-Software Foundation, write to the Free Software Foundation; we sometimes
-make exceptions for this.  Our decision will be guided by the two goals
-of preserving the free status of all derivatives of our free software and
-of promoting the sharing and reuse of software generally.
-
-			    NO WARRANTY
-
-  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
-FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
-OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
-PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
-OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
-MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
-TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
-PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
-REPAIR OR CORRECTION.
-
-  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
-WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
-REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
-INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
-OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
-TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
-YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
-PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
-POSSIBILITY OF SUCH DAMAGES.
-
-		     END OF TERMS AND CONDITIONS
-
-Note:
-Individual files contain the following tag instead of the full license text.
-
-    SPDX-License-Identifier: GPL-2.0-only
-
-This enables machine processing of license information based on the SPDX
-License Identifiers that are here available: http://spdx.org/licenses/

bitbake/LICENSE.MIT

@@ -1,25 +0,0 @@
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in
-all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
-
-Note:
-Individual files contain the following tag instead of the full license text.
-
-    SPDX-License-Identifier: MIT
-
-This enables machine processing of license information based on the SPDX
-License Identifiers that are here available: http://spdx.org/licenses/

bitbake/SECURITY.md

@@ -1,24 +0,0 @@
-How to Report a Potential Vulnerability?
-========================================
-
-If you would like to report a public issue (for example, one with a released
-CVE number), please report it using the
-[https://bugzilla.yoctoproject.org/enter_bug.cgi?product=Security Security Bugzilla].
-If you have a patch ready, submit it following the same procedure as any other
-patch as described in README.md.
-
-If you are dealing with a not-yet released or urgent issue, please send a
-message to security AT yoctoproject DOT org, including as many details as
-possible: the layer or software module affected, the recipe and its version,
-and any example code, if available.
-
-Branches maintained with security fixes
----------------------------------------
-
-See [https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS Stable release and LTS]
-for detailed info regarding the policies and maintenance of Stable branches.
-
-The [https://wiki.yoctoproject.org/wiki/Releases Release page] contains a list of all
-releases of the Yocto Project. Versions in grey are no longer actively maintained with
-security patches, but well-tested patches may still be accepted for them for
-significant issues.

bitbake/bin/bitbake

@@ -1,4 +1,6 @@
 #!/usr/bin/env python3
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
 # Copyright (C) 2003, 2004  Chris Larson
 # Copyright (C) 2003, 2004  Phil Blundell
@@ -7,8 +9,18 @@
 # Copyright (C) 2005        ROAD GmbH
 # Copyright (C) 2006        Richard Purdie
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import os
 import sys
@@ -26,7 +38,7 @@ from bb.main import bitbake_main, BitBakeConfigParameters, BBMainException
 if sys.getfilesystemencoding() != "utf-8":
     sys.exit("Please use a locale setting which supports UTF-8 (such as LANG=en_US.UTF-8).\nPython can't change the filesystem locale after loading so we need a UTF-8 when Python starts or things won't work.")
 
-__version__ = "1.46.0"
+__version__ = "1.36.0"
 
 if __name__ == "__main__":
     if __version__ != bb.__version__:

bitbake/bin/bitbake-diffsigs

@@ -1,16 +1,27 @@
 #!/usr/bin/env python3
 
-# bitbake-diffsigs / bitbake-dumpsig
-# BitBake task signature data dump and comparison utility
+# bitbake-diffsigs
+# BitBake task signature data comparison utility
 #
 # Copyright (C) 2012-2013, 2017 Intel Corporation
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import os
 import sys
 import warnings
+import fnmatch
 import argparse
 import logging
 import pickle
@@ -21,10 +32,7 @@ import bb.tinfoil
 import bb.siggen
 import bb.msg
 
-myname = os.path.basename(sys.argv[0])
-logger = bb.msg.logger_create(myname)
-
-is_dump = myname == 'bitbake-dumpsig'
+logger = bb.msg.logger_create('bitbake-diffsigs')
 
 def find_siginfo(tinfoil, pn, taskname, sigs=None):
     result = None
@@ -51,8 +59,8 @@ def find_siginfo(tinfoil, pn, taskname, sigs=None):
         sys.exit(2)
     return result
 
-def find_siginfo_task(bbhandler, pn, taskname, sig1=None, sig2=None):
-    """ Find the most recent signature files for the specified PN/task """
+def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None, color=False):
+    """ Find the most recent signature files for the specified PN/task and compare them """
 
     if not taskname.startswith('do_'):
         taskname = 'do_%s' % taskname
@@ -71,81 +79,73 @@ def find_siginfo_task(bbhandler, pn, taskname, sig1=None, sig2=None):
         latestfiles = [sigfiles[sig1], sigfiles[sig2]]
     else:
         filedates = find_siginfo(bbhandler, pn, taskname)
-        latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-2:]
+        latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-3:]
         if not latestfiles:
             logger.error('No sigdata files found matching %s %s' % (pn, taskname))
             sys.exit(1)
+        elif len(latestfiles) < 2:
+            logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (pn, taskname))
+            sys.exit(1)
 
-    return latestfiles
+    # Define recursion callback
+    def recursecb(key, hash1, hash2):
+        hashes = [hash1, hash2]
+        hashfiles = find_siginfo(bbhandler, key, None, hashes)
 
+        recout = []
+        if len(hashfiles) == 0:
+            recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2))
+        elif not hash1 in hashfiles:
+            recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash1))
+        elif not hash2 in hashfiles:
+            recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2))
+        else:
+            out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color)
+            for change in out2:
+                for line in change.splitlines():
+                    recout.append('  ' + line)
 
-# Define recursion callback
-def recursecb(key, hash1, hash2):
-    hashes = [hash1, hash2]
-    hashfiles = find_siginfo(tinfoil, key, None, hashes)
+        return recout
 
-    recout = []
-    if len(hashfiles) == 0:
-        recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2))
-    elif not hash1 in hashfiles:
-        recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash1))
-    elif not hash2 in hashfiles:
-        recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2))
-    else:
-        out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color)
-        for change in out2:
-            for line in change.splitlines():
-                recout.append('    ' + line)
+    # Recurse into signature comparison
+    logger.debug("Signature file (previous): %s" % latestfiles[-2])
+    logger.debug("Signature file (latest): %s" % latestfiles[-1])
+    output = bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb, color=color)
+    if output:
+        print('\n'.join(output))
+    sys.exit(0)
 
-    return recout
 
 
 parser = argparse.ArgumentParser(
-    description=("Dumps" if is_dump else "Compares") + " siginfo/sigdata files written out by BitBake")
+    description="Compares siginfo/sigdata files written out by BitBake")
 
-parser.add_argument('-D', '--debug',
+parser.add_argument('-d', '--debug',
                     help='Enable debug output',
                     action='store_true')
 
-if is_dump:
-    parser.add_argument("-t", "--task",
-            help="find the signature data file for the last run of the specified task",
-            action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname'))
+parser.add_argument('--color',
+        help='Colorize output (where %(metavar)s is %(choices)s)',
+        choices=['auto', 'always', 'never'], default='auto', metavar='color')
 
-    parser.add_argument("sigdatafile1",
-            help="Signature file to dump. Not used when using -t/--task.",
-            action="store", nargs='?', metavar="sigdatafile")
-else:
-    parser.add_argument('-c', '--color',
-            help='Colorize the output (where %(metavar)s is %(choices)s)',
-            choices=['auto', 'always', 'never'], default='auto', metavar='color')
+parser.add_argument("-t", "--task",
+        help="find the signature data files for last two runs of the specified task and compare them",
+        action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname'))
 
-    parser.add_argument('-d', '--dump',
-            help='Dump the last signature data instead of comparing (equivalent to using bitbake-dumpsig)',
-            action='store_true')
+parser.add_argument("-s", "--signature",
+        help="With -t/--task, specify the signatures to look for instead of taking the last two",
+        action="store", dest="sigargs", nargs=2, metavar=('fromsig', 'tosig'))
 
-    parser.add_argument("-t", "--task",
-            help="find the signature data files for the last two runs of the specified task and compare them",
-            action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname'))
+parser.add_argument("sigdatafile1",
+        help="First signature file to compare (or signature file to dump, if second not specified). Not used when using -t/--task.",
+        action="store", nargs='?')
 
-    parser.add_argument("-s", "--signature",
-            help="With -t/--task, specify the signatures to look for instead of taking the last two",
-            action="store", dest="sigargs", nargs=2, metavar=('fromsig', 'tosig'))
+parser.add_argument("sigdatafile2",
+        help="Second signature file to compare",
+        action="store", nargs='?')
 
-    parser.add_argument("sigdatafile1",
-            help="First signature file to compare (or signature file to dump, if second not specified). Not used when using -t/--task.",
-            action="store", nargs='?')
-
-    parser.add_argument("sigdatafile2",
-            help="Second signature file to compare",
-            action="store", nargs='?')
 
 options = parser.parse_args()
-if is_dump:
-    options.color = 'never'
-    options.dump = True
-    options.sigdatafile2 = None
-    options.sigargs = None
 
 if options.debug:
     logger.setLevel(logging.DEBUG)
@@ -155,32 +155,17 @@ color = (options.color == 'always' or (options.color == 'auto' and sys.stdout.is
 if options.taskargs:
     with bb.tinfoil.Tinfoil() as tinfoil:
         tinfoil.prepare(config_only=True)
-        if not options.dump and options.sigargs:
-            files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], options.sigargs[1])
+        if options.sigargs:
+            find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], options.sigargs[1], color=color)
         else:
-            files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1])
-
-        if options.dump:
-            logger.debug("Signature file: %s" % files[-1])
-            output = bb.siggen.dump_sigfile(files[-1])
-        else:
-            if len(files) < 2:
-                logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (options.taskargs[0], options.taskargs[1]))
-                sys.exit(1)
-
-            # Recurse into signature comparison
-            logger.debug("Signature file (previous): %s" % files[-2])
-            logger.debug("Signature file (latest): %s" % files[-1])
-            output = bb.siggen.compare_sigfiles(files[-2], files[-1], recursecb, color=color)
+            find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1], color=color)
 else:
     if options.sigargs:
         logger.error('-s/--signature can only be used together with -t/--task')
         sys.exit(1)
     try:
-        if not options.dump and options.sigdatafile1 and options.sigdatafile2:
-            with bb.tinfoil.Tinfoil() as tinfoil:
-                tinfoil.prepare(config_only=True)
-                output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, recursecb, color=color)
+        if options.sigdatafile1 and options.sigdatafile2:
+            output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, color=color)
         elif options.sigdatafile1:
             output = bb.siggen.dump_sigfile(options.sigdatafile1)
         else:
@@ -194,5 +179,5 @@ else:
         logger.error('Invalid signature data - ensure you are specifying sigdata/siginfo files')
         sys.exit(1)
 
-if output:
-    print('\n'.join(output))
+    if output:
+        print('\n'.join(output))

bitbake/bin/bitbake-dumpsig

@@ -1 +0,0 @@
-bitbake-diffsigs

bitbake/bin/bitbake-dumpsig

@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+
+# bitbake-dumpsig
+# BitBake task signature dump utility
+#
+# Copyright (C) 2013 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import sys
+import warnings
+import optparse
+import logging
+import pickle
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
+
+import bb.tinfoil
+import bb.siggen
+import bb.msg
+
+logger = bb.msg.logger_create('bitbake-dumpsig')
+
+def find_siginfo_task(bbhandler, pn, taskname):
+    """ Find the most recent signature file for the specified PN/task """
+
+    if not hasattr(bb.siggen, 'find_siginfo'):
+        logger.error('Metadata does not support finding signature data files')
+        sys.exit(1)
+
+    if not taskname.startswith('do_'):
+        taskname = 'do_%s' % taskname
+
+    filedates = bb.siggen.find_siginfo(pn, taskname, None, bbhandler.config_data)
+    latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-1:]
+    if not latestfiles:
+        logger.error('No sigdata files found matching %s %s' % (pn, taskname))
+        sys.exit(1)
+
+    return latestfiles[0]
+
+parser = optparse.OptionParser(
+    description = "Dumps siginfo/sigdata files written out by BitBake",
+    usage = """
+  %prog -t recipename taskname
+  %prog sigdatafile""")
+
+parser.add_option("-D", "--debug",
+        help = "enable debug",
+        action = "store_true", dest="debug", default = False)
+
+parser.add_option("-t", "--task",
+        help = "find the signature data file for the specified task",
+        action="store", dest="taskargs", nargs=2, metavar='recipename taskname')
+
+options, args = parser.parse_args(sys.argv)
+
+if options.debug:
+    logger.setLevel(logging.DEBUG)
+
+if options.taskargs:
+    tinfoil = bb.tinfoil.Tinfoil()
+    tinfoil.prepare(config_only = True)
+    file = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1])
+    logger.debug("Signature file: %s" % file)
+elif len(args) == 1:
+    parser.print_help()
+    sys.exit(0)
+else:
+    file = args[1]
+
+try:
+    output = bb.siggen.dump_sigfile(file)
+except IOError as e:
+    logger.error(str(e))
+    sys.exit(1)
+except (pickle.UnpicklingError, EOFError):
+    logger.error('Invalid signature data - ensure you are specifying a sigdata/siginfo file')
+    sys.exit(1)
+
+if output:
+    print('\n'.join(output))

bitbake/bin/bitbake-getvar

@@ -1,48 +0,0 @@
-#! /usr/bin/env python3
-#
-# Copyright (C) 2021 Richard Purdie
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
-
-import argparse
-import io
-import os
-import sys
-
-bindir = os.path.dirname(__file__)
-topdir = os.path.dirname(bindir)
-sys.path[0:0] = [os.path.join(topdir, 'lib')]
-
-import bb.tinfoil
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser(description="Bitbake Query Variable")
-    parser.add_argument("variable", help="variable name to query")
-    parser.add_argument("-r", "--recipe", help="Recipe name to query", default=None, required=False)
-    parser.add_argument('-u', '--unexpand', help='Do not expand the value (with --value)', action="store_true")
-    parser.add_argument('-f', '--flag', help='Specify a variable flag to query (with --value)', default=None)
-    parser.add_argument('--value', help='Only report the value, no history and no variable name', action="store_true")
-    args = parser.parse_args()
-
-    if args.unexpand and not args.value:
-        print("--unexpand only makes sense with --value")
-        sys.exit(1)
-
-    if args.flag and not args.value:
-        print("--flag only makes sense with --value")
-        sys.exit(1)
-
-    with bb.tinfoil.Tinfoil(tracking=True) as tinfoil:
-        if args.recipe:
-            tinfoil.prepare(quiet=2)
-            d = tinfoil.parse_recipe(args.recipe)
-        else:
-            tinfoil.prepare(quiet=2, config_only=True)
-            d = tinfoil.config_data
-        if args.flag:
-            print(str(d.getVarFlag(args.variable, args.flag, expand=(not args.unexpand))))
-        elif args.value:
-            print(str(d.getVar(args.variable, expand=(not args.unexpand))))
-        else:
-            bb.data.emit_var(args.variable, d=d, all=True)

bitbake/bin/bitbake-hashclient

@@ -1,170 +0,0 @@
-#! /usr/bin/env python3
-#
-# Copyright (C) 2019 Garmin Ltd.
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
-
-import argparse
-import hashlib
-import logging
-import os
-import pprint
-import sys
-import threading
-import time
-
-try:
-    import tqdm
-    ProgressBar = tqdm.tqdm
-except ImportError:
-    class ProgressBar(object):
-        def __init__(self, *args, **kwargs):
-            pass
-
-        def __enter__(self):
-            return self
-
-        def __exit__(self, *args, **kwargs):
-            pass
-
-        def update(self):
-            pass
-
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib'))
-
-import hashserv
-
-DEFAULT_ADDRESS = 'unix://./hashserve.sock'
-METHOD = 'stress.test.method'
-
-
-def main():
-    def handle_stats(args, client):
-        if args.reset:
-            s = client.reset_stats()
-        else:
-            s = client.get_stats()
-        pprint.pprint(s)
-        return 0
-
-    def handle_stress(args, client):
-        def thread_main(pbar, lock):
-            nonlocal found_hashes
-            nonlocal missed_hashes
-            nonlocal max_time
-
-            client = hashserv.create_client(args.address)
-
-            for i in range(args.requests):
-                taskhash = hashlib.sha256()
-                taskhash.update(args.taskhash_seed.encode('utf-8'))
-                taskhash.update(str(i).encode('utf-8'))
-
-                start_time = time.perf_counter()
-                l = client.get_unihash(METHOD, taskhash.hexdigest())
-                elapsed = time.perf_counter() - start_time
-
-                with lock:
-                    if l:
-                        found_hashes += 1
-                    else:
-                        missed_hashes += 1
-
-                    max_time = max(elapsed, max_time)
-                    pbar.update()
-
-        max_time = 0
-        found_hashes = 0
-        missed_hashes = 0
-        lock = threading.Lock()
-        total_requests = args.clients * args.requests
-        start_time = time.perf_counter()
-        with ProgressBar(total=total_requests) as pbar:
-            threads = [threading.Thread(target=thread_main, args=(pbar, lock), daemon=False) for _ in range(args.clients)]
-            for t in threads:
-                t.start()
-
-            for t in threads:
-                t.join()
-
-        elapsed = time.perf_counter() - start_time
-        with lock:
-            print("%d requests in %.1fs. %.1f requests per second" % (total_requests, elapsed, total_requests / elapsed))
-            print("Average request time %.8fs" % (elapsed / total_requests))
-            print("Max request time was %.8fs" % max_time)
-            print("Found %d hashes, missed %d" % (found_hashes, missed_hashes))
-
-        if args.report:
-            with ProgressBar(total=args.requests) as pbar:
-                for i in range(args.requests):
-                    taskhash = hashlib.sha256()
-                    taskhash.update(args.taskhash_seed.encode('utf-8'))
-                    taskhash.update(str(i).encode('utf-8'))
-
-                    outhash = hashlib.sha256()
-                    outhash.update(args.outhash_seed.encode('utf-8'))
-                    outhash.update(str(i).encode('utf-8'))
-
-                    client.report_unihash(taskhash.hexdigest(), METHOD, outhash.hexdigest(), taskhash.hexdigest())
-
-                    with lock:
-                        pbar.update()
-
-    parser = argparse.ArgumentParser(description='Hash Equivalence Client')
-    parser.add_argument('--address', default=DEFAULT_ADDRESS, help='Server address (default "%(default)s")')
-    parser.add_argument('--log', default='WARNING', help='Set logging level')
-
-    subparsers = parser.add_subparsers()
-
-    stats_parser = subparsers.add_parser('stats', help='Show server stats')
-    stats_parser.add_argument('--reset', action='store_true',
-                              help='Reset server stats')
-    stats_parser.set_defaults(func=handle_stats)
-
-    stress_parser = subparsers.add_parser('stress', help='Run stress test')
-    stress_parser.add_argument('--clients', type=int, default=10,
-                               help='Number of simultaneous clients')
-    stress_parser.add_argument('--requests', type=int, default=1000,
-                               help='Number of requests each client will perform')
-    stress_parser.add_argument('--report', action='store_true',
-                               help='Report new hashes')
-    stress_parser.add_argument('--taskhash-seed', default='',
-                               help='Include string in taskhash')
-    stress_parser.add_argument('--outhash-seed', default='',
-                               help='Include string in outhash')
-    stress_parser.set_defaults(func=handle_stress)
-
-    args = parser.parse_args()
-
-    logger = logging.getLogger('hashserv')
-
-    level = getattr(logging, args.log.upper(), None)
-    if not isinstance(level, int):
-        raise ValueError('Invalid log level: %s' % args.log)
-
-    logger.setLevel(level)
-    console = logging.StreamHandler()
-    console.setLevel(level)
-    logger.addHandler(console)
-
-    func = getattr(args, 'func', None)
-    if func:
-        client = hashserv.create_client(args.address)
-        # Try to establish a connection to the server now to detect failures
-        # early
-        client.connect()
-
-        return func(args, client)
-
-    return 0
-
-
-if __name__ == '__main__':
-    try:
-        ret = main()
-    except Exception:
-        ret = 1
-        import traceback
-        traceback.print_exc()
-    sys.exit(ret)

bitbake/bin/bitbake-hashserv

@@ -1,62 +0,0 @@
-#! /usr/bin/env python3
-#
-# Copyright (C) 2018 Garmin Ltd.
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
-
-import os
-import sys
-import logging
-import argparse
-import sqlite3
-
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib'))
-
-import hashserv
-
-VERSION = "1.0.0"
-
-DEFAULT_BIND = 'unix://./hashserve.sock'
-
-
-def main():
-    parser = argparse.ArgumentParser(description='Hash Equivalence Reference Server. Version=%s' % VERSION,
-                                     epilog='''The bind address is the path to a unix domain socket if it is
-                                               prefixed with "unix://". Otherwise, it is an IP address
-                                               and port in form ADDRESS:PORT. To bind to all addresses, leave
-                                               the ADDRESS empty, e.g. "--bind :8686". To bind to a specific
-                                               IPv6 address, enclose the address in "[]", e.g.
-                                               "--bind [::1]:8686"'''
-                                     )
-
-    parser.add_argument('--bind', default=DEFAULT_BIND, help='Bind address (default "%(default)s")')
-    parser.add_argument('--database', default='./hashserv.db', help='Database file (default "%(default)s")')
-    parser.add_argument('--log', default='WARNING', help='Set logging level')
-
-    args = parser.parse_args()
-
-    logger = logging.getLogger('hashserv')
-
-    level = getattr(logging, args.log.upper(), None)
-    if not isinstance(level, int):
-        raise ValueError('Invalid log level: %s' % args.log)
-
-    logger.setLevel(level)
-    console = logging.StreamHandler()
-    console.setLevel(level)
-    logger.addHandler(console)
-
-    server = hashserv.create_server(args.bind, args.database)
-    server.serve_forever()
-    return 0
-
-
-if __name__ == '__main__':
-    try:
-        ret = main()
-    except Exception:
-        ret = 1
-        import traceback
-        traceback.print_exc()
-    sys.exit(ret)

bitbake/bin/bitbake-layers

@@ -7,8 +7,18 @@
 # Copyright (C) 2011 Mentor Graphics Corporation
 # Copyright (C) 2011-2015 Intel Corporation
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import logging
 import os
@@ -52,9 +62,7 @@ def main():
 
     # Need to re-run logger_create with color argument
     # (will be the same logger since it has the same name)
-    bb.msg.logger_create('bitbake-layers', output=sys.stdout,
-                         color=global_args.color,
-                         level=logger.getEffectiveLevel())
+    bb.msg.logger_create('bitbake-layers', output=sys.stdout, color=global_args.color)
 
     plugins = []
     tinfoil = bb.tinfoil.Tinfoil(tracking=True)

bitbake/bin/bitbake-prserv

@@ -1,8 +1,4 @@
 #!/usr/bin/env python3
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
-
 import os
 import sys,logging
 import optparse

bitbake/bin/bitbake-selftest

@@ -2,8 +2,18 @@
 #
 # Copyright (C) 2012 Richard Purdie
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import os
 import sys, logging
@@ -12,25 +22,16 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib
 import unittest
 try:
     import bb
-    import hashserv
-    import layerindexlib
 except RuntimeError as exc:
     sys.exit(str(exc))
 
 tests = ["bb.tests.codeparser",
-         "bb.tests.cooker",
          "bb.tests.cow",
          "bb.tests.data",
          "bb.tests.event",
          "bb.tests.fetch",
          "bb.tests.parse",
-         "bb.tests.persist_data",
-         "bb.tests.runqueue",
-         "bb.tests.utils",
-         "hashserv.tests",
-         "layerindexlib.tests.layerindexobj",
-         "layerindexlib.tests.restapi",
-         "layerindexlib.tests.cooker"]
+         "bb.tests.utils"]
 
 for t in tests:
     t = '.'.join(t.split('.')[:3])

bitbake/bin/bitbake-worker

@@ -1,7 +1,4 @@
 #!/usr/bin/env python3
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
 
 import os
 import sys
@@ -65,6 +62,7 @@ if 0:
     format_str = "%(levelname)s: %(message)s"
     conlogformat = bb.msg.BBLogFormatter(format_str)
     consolelog = logging.FileHandler(logfilename)
+    bb.msg.addDefaultlogFilter(consolelog)
     consolelog.setFormatter(conlogformat)
     logger.addHandler(consolelog)
 
@@ -138,7 +136,7 @@ def sigterm_handler(signum, frame):
     os.killpg(0, signal.SIGTERM)
     sys.exit()
 
-def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, taskhash, unihash, appends, taskdepdata, extraconfigdata, quieterrors=False, dry_run_exec=False):
+def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, appends, taskdepdata, extraconfigdata, quieterrors=False, dry_run_exec=False):
     # We need to setup the environment BEFORE the fork, since
     # a fork() or exec*() activates PSEUDO...
 
@@ -194,6 +192,9 @@ def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, taskha
             global worker_pipe_lock
             pipein.close()
 
+            signal.signal(signal.SIGTERM, sigterm_handler)
+            # Let SIGHUP exit as SIGTERM
+            signal.signal(signal.SIGHUP, sigterm_handler)
             bb.utils.signal_on_parent_exit("SIGTERM")
 
             # Save out the PID so that the event can include it the
@@ -208,11 +209,6 @@ def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, taskha
             # This ensures signals sent to the controlling terminal like Ctrl+C
             # don't stop the child processes.
             os.setsid()
-
-            signal.signal(signal.SIGTERM, sigterm_handler)
-            # Let SIGHUP exit as SIGTERM
-            signal.signal(signal.SIGHUP, sigterm_handler)
-
             # No stdin
             newsi = os.open(os.devnull, os.O_RDWR)
             os.dup2(newsi, sys.stdin.fileno())
@@ -235,13 +231,10 @@ def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, taskha
                     the_data.setVar(varname, value)
 
                 bb.parse.siggen.set_taskdata(workerdata["sigdata"])
-                if "newhashes" in workerdata:
-                    bb.parse.siggen.set_taskhashes(workerdata["newhashes"])
                 ret = 0
 
                 the_data = bb_cache.loadDataFull(fn, appends)
-                the_data.setVar('BB_TASKHASH', taskhash)
-                the_data.setVar('BB_UNIHASH', unihash)
+                the_data.setVar('BB_TASKHASH', workerdata["runq_hash"][task])
 
                 bb.utils.set_process_name("%s:%s" % (the_data.getVar("PN"), taskname.replace("do_", "")))
 
@@ -380,7 +373,6 @@ class BitbakeWorker(object):
                 self.handle_item(b"cookerconfig", self.handle_cookercfg)
                 self.handle_item(b"extraconfigdata", self.handle_extraconfigdata)
                 self.handle_item(b"workerdata", self.handle_workerdata)
-                self.handle_item(b"newtaskhashes", self.handle_newtaskhashes)
                 self.handle_item(b"runtask", self.handle_runtask)
                 self.handle_item(b"finishnow", self.handle_finishnow)
                 self.handle_item(b"ping", self.handle_ping)
@@ -413,16 +405,12 @@ class BitbakeWorker(object):
 
     def handle_workerdata(self, data):
         self.workerdata = pickle.loads(data)
-        bb.build.verboseShellLogging = self.workerdata["build_verbose_shell"]
-        bb.build.verboseStdoutLogging = self.workerdata["build_verbose_stdout"]
-        bb.msg.loggerDefaultLogLevel = self.workerdata["logdefaultlevel"]
+        bb.msg.loggerDefaultDebugLevel = self.workerdata["logdefaultdebug"]
+        bb.msg.loggerDefaultVerbose = self.workerdata["logdefaultverbose"]
+        bb.msg.loggerVerboseLogs = self.workerdata["logdefaultverboselogs"]
         bb.msg.loggerDefaultDomains = self.workerdata["logdefaultdomain"]
         for mc in self.databuilder.mcdata:
             self.databuilder.mcdata[mc].setVar("PRSERV_HOST", self.workerdata["prhost"])
-            self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", self.workerdata["hashservaddr"])
-
-    def handle_newtaskhashes(self, data):
-        self.workerdata["newhashes"] = pickle.loads(data)
 
     def handle_ping(self, _):
         workerlog_write("Handling ping\n")
@@ -437,10 +425,10 @@ class BitbakeWorker(object):
         sys.exit(0)
 
     def handle_runtask(self, data):
-        fn, task, taskname, taskhash, unihash, quieterrors, appends, taskdepdata, dry_run_exec = pickle.loads(data)
+        fn, task, taskname, quieterrors, appends, taskdepdata, dry_run_exec = pickle.loads(data)
         workerlog_write("Handling runtask %s %s %s\n" % (task, fn, taskname))
 
-        pid, pipein, pipeout = fork_off_task(self.cookercfg, self.data, self.databuilder, self.workerdata, fn, task, taskname, taskhash, unihash, appends, taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec)
+        pid, pipein, pipeout = fork_off_task(self.cookercfg, self.data, self.databuilder, self.workerdata, fn, task, taskname, appends, taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec)
 
         self.build_pids[pid] = task
         self.build_pipes[pid] = runQueueWorkerPipe(pipein, pipeout)
@@ -505,11 +493,9 @@ except BaseException as e:
         import traceback
         sys.stderr.write(traceback.format_exc())
         sys.stderr.write(str(e))
-finally:
-    worker_thread_exit = True
-    worker_thread.join()
 
-workerlog_write("exiting")
-if not normalexit:
-    sys.exit(1)
+worker_thread_exit = True
+worker_thread.join()
+
+workerlog_write("exitting")
 sys.exit(0)

bitbake/bin/bitdoc

@@ -1,9 +1,21 @@
 #!/usr/bin/env python3
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
 # Copyright (C) 2005 Holger Hans Peter Freyther
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import optparse, os, sys
 

bitbake/bin/git-make-shallow

@@ -1,8 +1,4 @@
 #!/usr/bin/env python3
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
-
 """git-make-shallow: make the current git repository shallow
 
 Remove the history of the specified revisions, then optionally filter the

bitbake/bin/toaster

@@ -3,18 +3,25 @@
 # toaster - shell script to start Toaster
 
 # Copyright (C) 2013-2015 Intel Corp.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
 #
-# SPDX-License-Identifier: GPL-2.0-or-later
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
 #
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see http://www.gnu.org/licenses/.
 
 HELP="
-Usage 1: source toaster start|stop [webport=<address:port>] [noweb] [nobuild] [toasterdir]
+Usage: source toaster start|stop [webport=<address:port>] [noweb]
     Optional arguments:
-        [nobuild] Setup the environment for capturing builds with toaster but disable managed builds
-        [noweb] Setup the environment for capturing builds with toaster but don't start the web server
+        [noweb] Setup the environment for building with toaster but don't start the development server
         [webport] Set the development server (default: localhost:8000)
-        [toasterdir] Set absolute path to be used as TOASTER_DIR (default: BUILDDIR/../)
-Usage 2: source toaster manage [createsuperuser|lsupdates|migrate|makemigrations|checksettings|collectstatic|...]
 "
 
 custom_extention()
@@ -60,7 +67,7 @@ webserverKillAll()
         if [ -f ${pidfile} ]; then
             pid=`cat ${pidfile}`
             while kill -0 $pid 2>/dev/null; do
-                kill -SIGTERM $pid 2>/dev/null
+                kill -SIGTERM -$pid 2>/dev/null
                 sleep 1
             done
             rm  ${pidfile}
@@ -83,7 +90,7 @@ webserverStartAll()
 
     echo "Starting webserver..."
 
-    $MANAGE runserver --noreload "$ADDR_PORT" \
+    $MANAGE runserver "$ADDR_PORT" \
            </dev/null >>${BUILDDIR}/toaster_web.log 2>&1 \
            & echo $! >${BUILDDIR}/.toastermain.pid
 
@@ -152,9 +159,7 @@ fi
 
 export BBBASEDIR=`dirname $TOASTER`/..
 MANAGE="python3 $BBBASEDIR/lib/toaster/manage.py"
-if [ -z "$OE_ROOT" ]; then
-    OE_ROOT=`dirname $TOASTER`/../..
-fi
+OE_ROOT=`dirname $TOASTER`/../..
 
 # this is the configuraton file we are using for toaster
 # we are using the same logic that oe-setup-builddir uses
@@ -178,18 +183,13 @@ unset OE_ROOT
 
 
 WEBSERVER=1
-export TOASTER_BUILDSERVER=1
 ADDR_PORT="localhost:8000"
-TOASTERDIR=`dirname $BUILDDIR`
 unset CMD
 for param in $*; do
     case $param in
     noweb )
             WEBSERVER=0
     ;;
-    nobuild )
-            TOASTER_BUILDSERVER=0
-    ;;
     start )
             CMD=$param
     ;;
@@ -206,24 +206,13 @@ for param in $*; do
                 ADDR_PORT="localhost:$PORT"
             fi
     ;;
-    toasterdir=*)
-            TOASTERDIR="${param#*=}"
-    ;;
-    manage )
-            CMD=$param
-            manage_cmd=""
-    ;;
     --help)
             echo "$HELP"
             return 0
     ;;
     *)
-            if [ "manage" == "$CMD" ] ; then
-                manage_cmd="$manage_cmd $param"
-            else
-                echo "$HELP"
-                exit 1
-            fi
+            echo "$HELP"
+            return 1
     ;;
 
     esac
@@ -247,7 +236,7 @@ fi
 # 2) the build dir (in build)
 # 3) the sqlite db if that is being used.
 # 4) pid's we need to clean up on exit/shutdown
-export TOASTER_DIR=$TOASTERDIR
+export TOASTER_DIR=`dirname $BUILDDIR`
 export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE TOASTER_DIR"
 
 # Determine the action. If specified by arguments, fine, if not, toggle it
@@ -297,13 +286,9 @@ case $CMD in
             return 4
         fi
         export BITBAKE_UI='toasterui'
-        if [ $TOASTER_BUILDSERVER -eq 1 ] ; then
-            $MANAGE runbuilds \
-               </dev/null >>${BUILDDIR}/toaster_runbuilds.log 2>&1 \
-               & echo $! >${BUILDDIR}/.runbuilds.pid
-        else
-            echo "Toaster build server not started."
-        fi
+        $MANAGE runbuilds \
+           </dev/null >>${BUILDDIR}/toaster_runbuilds.log 2>&1 \
+           & echo $! >${BUILDDIR}/.runbuilds.pid
 
         # set fail safe stop system on terminal exit
         trap stop_system SIGHUP
@@ -315,10 +300,6 @@ case $CMD in
         stop_system
         echo "Successful ${CMD}."
     ;;
-    manage )
-        cd $BBBASEDIR/lib/toaster
-        $MANAGE $manage_cmd
-    ;;
 esac
 custom_extention toaster_postpend $CMD $ADDR_PORT
 

bitbake/bin/toaster-eventreplay

@@ -1,12 +1,25 @@
 #!/usr/bin/env python3
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
 # Copyright (C) 2014        Alex Damian
 #
-# SPDX-License-Identifier: GPL-2.0-only
-#
 # This file re-uses code spread throughout other Bitbake source files.
 # As such, all other copyrights belong to their own right holders.
 #
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 """
 This command takes a filename as a single parameter. The filename is read

bitbake/contrib/autobuilderlog.json

@@ -1,13 +0,0 @@
-{
-    "version": 1,
-    "loggers": {
-        "BitBake.SigGen.HashEquiv": {
-            "level": "VERBOSE",
-            "handlers": ["BitBake.verbconsole"]
-        },
-        "BitBake.RunQueue.HashEquiv": {
-            "level": "VERBOSE",
-            "handlers": ["BitBake.verbconsole"]
-        }
-    }
-}

bitbake/contrib/dump_cache.py

@@ -1,6 +1,8 @@
 #!/usr/bin/env python3
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
-# Copyright (C) 2012, 2018 Wind River Systems, Inc.
+# Copyright (C) 2012 Wind River Systems, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 as
@@ -16,68 +18,51 @@
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 #
-# Used for dumping the bb_cache.dat
+# This is used for dumping the bb_cache.dat, the output format is:
+# recipe_path PN PV PACKAGES
 #
 import os
 import sys
-import argparse
+import warnings
 
 # For importing bb.cache
 sys.path.insert(0, os.path.join(os.path.abspath(os.path.dirname(sys.argv[0])), '../lib'))
 from bb.cache import CoreRecipeInfo
 
-import pickle
+import pickle as pickle
 
-class DumpCache(object):
-    def __init__(self):
-        parser = argparse.ArgumentParser(
-            description="bb_cache.dat's dumper",
-            epilog="Use %(prog)s --help to get help")
-        parser.add_argument("-r", "--recipe",
-            help="specify the recipe, default: all recipes", action="store")
-        parser.add_argument("-m", "--members",
-            help = "specify the member, use comma as separator for multiple ones, default: all members", action="store", default="")
-        parser.add_argument("-s", "--skip",
-            help = "skip skipped recipes", action="store_true")
-        parser.add_argument("cachefile",
-            help = "specify bb_cache.dat", nargs = 1, action="store", default="")
+def main(argv=None):
+    """
+    Get the mapping for the target recipe.
+    """
+    if len(argv) != 1:
+        print("Error, need one argument!", file=sys.stderr)
+        return 2
 
-        self.args = parser.parse_args()
+    cachefile = argv[0]
 
-    def main(self):
-        with open(self.args.cachefile[0], "rb") as cachefile:
-            pickled = pickle.Unpickler(cachefile)
-            while True:
-                try:
-                    key = pickled.load()
-                    val = pickled.load()
-                except Exception:
-                    break
-                if isinstance(val, CoreRecipeInfo):
-                    pn = val.pn
+    with open(cachefile, "rb") as cachefile:
+        pickled = pickle.Unpickler(cachefile)
+        while cachefile:
+            try:
+                key = pickled.load()
+                val = pickled.load()
+            except Exception:
+                break
+            if isinstance(val, CoreRecipeInfo) and (not val.skipped):
+                pn = val.pn
+                # Filter out the native recipes.
+                if key.startswith('virtual:native:') or pn.endswith("-native"):
+                    continue
 
-                    if self.args.recipe and self.args.recipe != pn:
-                        continue
+                # 1.0 is the default version for a no PV recipe.
+                if "pv" in val.__dict__:
+                    pv = val.pv
+                else:
+                    pv = "1.0"
 
-                    if self.args.skip and val.skipped:
-                        continue
-
-                    if self.args.members:
-                        out = key
-                        for member in self.args.members.split(','):
-                            out += ": %s" % val.__dict__.get(member)
-                        print("%s" % out)
-                    else:
-                        print("%s: %s" % (key, val.__dict__))
-                elif not self.args.recipe:
-                    print("%s %s" % (key, val))
+                print("%s %s %s %s" % (key, pn, pv, ' '.join(val.packages)))
 
 if __name__ == "__main__":
-    try:
-        dump = DumpCache()
-        ret = dump.main()
-    except Exception as esc:
-        ret = 1
-        import traceback
-        traceback.print_exc()
-    sys.exit(ret)
+    sys.exit(main(sys.argv[1:]))
+

bitbake/contrib/vim/indent/bitbake.vim

@@ -1,343 +0,0 @@
-" Vim indent file
-" Language:             BitBake
-" Copyright:            Copyright (C) 2019 Agilent Technologies, Inc.
-" Maintainer:           Chris Laplante <chris.laplante@agilent.com>
-" License:              You may redistribute this under the same terms as Vim itself
-
-
-if exists("b:did_indent")
-    finish
-endif
-
-if exists("*BitbakeIndent")
-    finish
-endif
-
-runtime! indent/sh.vim
-unlet b:did_indent
-
-setlocal indentexpr=BitbakeIndent(v:lnum)
-setlocal autoindent nolisp
-
-function s:is_bb_python_func_def(lnum)
-    let stack = synstack(a:lnum, 1)
-    if len(stack) == 0
-        return 0
-    endif
-
-    let top = synIDattr(stack[0], "name")
-    echo top
-
-    return synIDattr(stack[0], "name") == "bbPyFuncDef"
-endfunction
-
-"""" begin modified from indent/python.vim, upstream commit 7a9bd7c1e0ce1baf5a02daf36eeae3638aa315c7
-"""" This copied code is licensed the same as Vim itself.
-setlocal indentkeys+=<:>,=elif,=except
-
-let s:keepcpo= &cpo
-set cpo&vim
-
-let s:maxoff = 50	" maximum number of lines to look backwards for ()
-
-function GetPythonIndent(lnum)
-
-  " If this line is explicitly joined: If the previous line was also joined,
-  " line it up with that one, otherwise add two 'shiftwidth'
-  if getline(a:lnum - 1) =~ '\\$'
-    if a:lnum > 1 && getline(a:lnum - 2) =~ '\\$'
-      return indent(a:lnum - 1)
-    endif
-    return indent(a:lnum - 1) + (exists("g:pyindent_continue") ? eval(g:pyindent_continue) : (shiftwidth() * 2))
-  endif
-
-  " If the start of the line is in a string don't change the indent.
-  if has('syntax_items')
-	\ && synIDattr(synID(a:lnum, 1, 1), "name") =~ "String$"
-    return -1
-  endif
-
-  " Search backwards for the previous non-empty line.
-  let plnum = prevnonblank(v:lnum - 1)
-
-  if plnum == 0
-    " This is the first non-empty line, use zero indent.
-    return 0
-  endif
-
-  call cursor(plnum, 1)
-
-  " Identing inside parentheses can be very slow, regardless of the searchpair()
-  " timeout, so let the user disable this feature if he doesn't need it
-  let disable_parentheses_indenting = get(g:, "pyindent_disable_parentheses_indenting", 0)
-
-  if disable_parentheses_indenting == 1
-    let plindent = indent(plnum)
-    let plnumstart = plnum
-  else
-    " searchpair() can be slow sometimes, limit the time to 150 msec or what is
-    " put in g:pyindent_searchpair_timeout
-    let searchpair_stopline = 0
-    let searchpair_timeout = get(g:, 'pyindent_searchpair_timeout', 150)
-
-    " If the previous line is inside parenthesis, use the indent of the starting
-    " line.
-    " Trick: use the non-existing "dummy" variable to break out of the loop when
-    " going too far back.
-    let parlnum = searchpair('(\|{\|\[', '', ')\|}\|\]', 'nbW',
-            \ "line('.') < " . (plnum - s:maxoff) . " ? dummy :"
-            \ . " synIDattr(synID(line('.'), col('.'), 1), 'name')"
-            \ . " =~ '\\(Comment\\|Todo\\|String\\)$'",
-            \ searchpair_stopline, searchpair_timeout)
-    if parlnum > 0
-      " We may have found the opening brace of a BitBake Python task, e.g. 'python do_task {'
-      " If so, ignore it here - it will be handled later.
-      if s:is_bb_python_func_def(parlnum)
-        let parlnum = 0
-        let plindent = indent(plnum)
-        let plnumstart = plnum
-      else
-        let plindent = indent(parlnum)
-        let plnumstart = parlnum
-      endif
-    else
-      let plindent = indent(plnum)
-      let plnumstart = plnum
-    endif
-
-    " When inside parenthesis: If at the first line below the parenthesis add
-    " two 'shiftwidth', otherwise same as previous line.
-    " i = (a
-    "       + b
-    "       + c)
-    call cursor(a:lnum, 1)
-    let p = searchpair('(\|{\|\[', '', ')\|}\|\]', 'bW',
-            \ "line('.') < " . (a:lnum - s:maxoff) . " ? dummy :"
-            \ . " synIDattr(synID(line('.'), col('.'), 1), 'name')"
-            \ . " =~ '\\(Comment\\|Todo\\|String\\)$'",
-            \ searchpair_stopline, searchpair_timeout)
-    if p > 0
-        if s:is_bb_python_func_def(p)
-          " Handle first non-empty line inside a BB Python task
-          if p == plnum
-              return shiftwidth()
-          endif
-
-          " Handle the user actually trying to close a BitBake Python task
-          let line = getline(a:lnum)
-          if line =~ '^\s*}'
-              return -2
-          endif
-
-          " Otherwise ignore the brace
-          let p = 0
-        else
-          if p == plnum
-            " When the start is inside parenthesis, only indent one 'shiftwidth'.
-            let pp = searchpair('(\|{\|\[', '', ')\|}\|\]', 'bW',
-                \ "line('.') < " . (a:lnum - s:maxoff) . " ? dummy :"
-                \ . " synIDattr(synID(line('.'), col('.'), 1), 'name')"
-                \ . " =~ '\\(Comment\\|Todo\\|String\\)$'",
-                \ searchpair_stopline, searchpair_timeout)
-            if pp > 0
-              return indent(plnum) + (exists("g:pyindent_nested_paren") ? eval(g:pyindent_nested_paren) : shiftwidth())
-            endif
-            return indent(plnum) + (exists("g:pyindent_open_paren") ? eval(g:pyindent_open_paren) : (shiftwidth() * 2))
-          endif
-          if plnumstart == p
-            return indent(plnum)
-          endif
-          return plindent
-        endif
-    endif
-
-  endif
-
-
-  " Get the line and remove a trailing comment.
-  " Use syntax highlighting attributes when possible.
-  let pline = getline(plnum)
-  let pline_len = strlen(pline)
-  if has('syntax_items')
-    " If the last character in the line is a comment, do a binary search for
-    " the start of the comment.  synID() is slow, a linear search would take
-    " too long on a long line.
-    if synIDattr(synID(plnum, pline_len, 1), "name") =~ "\\(Comment\\|Todo\\)$"
-      let min = 1
-      let max = pline_len
-      while min < max
-	let col = (min + max) / 2
-	if synIDattr(synID(plnum, col, 1), "name") =~ "\\(Comment\\|Todo\\)$"
-	  let max = col
-	else
-	  let min = col + 1
-	endif
-      endwhile
-      let pline = strpart(pline, 0, min - 1)
-    endif
-  else
-    let col = 0
-    while col < pline_len
-      if pline[col] == '#'
-	let pline = strpart(pline, 0, col)
-	break
-      endif
-      let col = col + 1
-    endwhile
-  endif
-
-  " If the previous line ended with a colon, indent this line
-  if pline =~ ':\s*$'
-    return plindent + shiftwidth()
-  endif
-
-  " If the previous line was a stop-execution statement...
-  " TODO: utilize this logic to deindent when ending a bbPyDefRegion
-  if getline(plnum) =~ '^\s*\(break\|continue\|raise\|return\|pass\|bb\.fatal\)\>'
-    " See if the user has already dedented
-    if indent(a:lnum) > indent(plnum) - shiftwidth()
-      " If not, recommend one dedent
-      return indent(plnum) - shiftwidth()
-    endif
-    " Otherwise, trust the user
-    return -1
-  endif
-
-  " If the current line begins with a keyword that lines up with "try"
-  if getline(a:lnum) =~ '^\s*\(except\|finally\)\>'
-    let lnum = a:lnum - 1
-    while lnum >= 1
-      if getline(lnum) =~ '^\s*\(try\|except\)\>'
-	let ind = indent(lnum)
-	if ind >= indent(a:lnum)
-	  return -1	" indent is already less than this
-	endif
-	return ind	" line up with previous try or except
-      endif
-      let lnum = lnum - 1
-    endwhile
-    return -1		" no matching "try"!
-  endif
-
-  " If the current line begins with a header keyword, dedent
-  if getline(a:lnum) =~ '^\s*\(elif\|else\)\>'
-
-    " Unless the previous line was a one-liner
-    if getline(plnumstart) =~ '^\s*\(for\|if\|try\)\>'
-      return plindent
-    endif
-
-    " Or the user has already dedented
-    if indent(a:lnum) <= plindent - shiftwidth()
-      return -1
-    endif
-
-    return plindent - shiftwidth()
-  endif
-
-  " When after a () construct we probably want to go back to the start line.
-  " a = (b
-  "       + c)
-  " here
-  if parlnum > 0
-    return plindent
-  endif
-
-  return -1
-
-endfunction
-
-let &cpo = s:keepcpo
-unlet s:keepcpo
-
-""" end of stuff from indent/python.vim
-
-
-let b:did_indent = 1
-setlocal indentkeys+=0\"
-
-
-function BitbakeIndent(lnum)
-    if !has('syntax_items')
-        return -1
-    endif
-
-    let stack = synstack(a:lnum, 1)
-    if len(stack) == 0
-        return -1
-    endif
-
-    let name = synIDattr(stack[0], "name")
-
-    " TODO: support different styles of indentation for assignments. For now,
-    " we only support like this:
-    " VAR = " \
-    "     value1 \
-    "     value2 \
-    " "
-    "
-    " i.e. each value indented by shiftwidth(), with the final quote " completely unindented.
-    if name == "bbVarValue"
-        " Quote handling is tricky. kernel.bbclass has this line for instance:
-        "     EXTRA_OEMAKE = " HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" " HOSTCPP="${BUILD_CPP}""
-        " Instead of trying to handle crazy cases like that, just assume that a
-        " double-quote on a line by itself (following an assignment) means the
-        " user is closing the assignment, and de-dent.
-        if getline(a:lnum) =~ '^\s*"$'
-            return 0
-        endif
-
-        let prevstack = synstack(a:lnum - 1, 1)
-        if len(prevstack) == 0
-            return -1
-        endif
-
-        let prevname = synIDattr(prevstack[0], "name")
-
-        " Only indent if there was actually a continuation character on
-        " the previous line, to avoid misleading indentation.
-        let prevlinelastchar = synIDattr(synID(a:lnum - 1, col([a:lnum - 1, "$"]) - 1, 1), "name")
-        let prev_continued = prevlinelastchar == "bbContinue"
-
-        " Did the previous line introduce an assignment?
-        if index(["bbVarDef", "bbVarFlagDef"], prevname) != -1
-            if prev_continued
-                return shiftwidth()
-            endif
-        endif
-
-        if !prev_continued
-            return 0
-        endif
-
-        " Autoindent can take it from here
-        return -1
-    endif
-
-    if index(["bbPyDefRegion", "bbPyFuncRegion"], name) != -1
-        let ret = GetPythonIndent(a:lnum)
-        " Should normally always be indented by at least one shiftwidth; but allow
-        " return of -1 (defer to autoindent) or -2 (force indent to 0)
-        if ret == 0
-            return shiftwidth()
-        elseif ret == -2
-            return 0
-        endif
-        return ret
-    endif
-
-    " TODO: GetShIndent doesn't detect tasks prepended with 'fakeroot'
-    " Need to submit a patch upstream to Vim to provide an extension point.
-    " Unlike the Python indenter, the Sh indenter is way too large to copy and
-    " modify here.
-    if name == "bbShFuncRegion"
-        return GetShIndent()
-    endif
-
-    " TODO:
-    "   + heuristics for de-denting out of a bbPyDefRegion? e.g. when the user
-    "       types an obvious BB keyword like addhandler or addtask, or starts
-    "       writing a shell task. Maybe too hard to implement...
-
-    return -1
-endfunction

bitbake/doc/.gitignore

@@ -1 +0,0 @@
-_build/

bitbake/doc/Makefile

@@ -1,35 +1,91 @@
-# Minimal makefile for Sphinx documentation
+# This is a single Makefile to handle all generated BitBake documents.
+# The Makefile needs to live in the documentation directory and all figures used
+# in any manuals must be .PNG files and live in the individual book's figures
+# directory.
+#
+# The Makefile has these targets:
+#
+#    pdf:      generates a PDF version of a manual.
+#    html:     generates an HTML version of a manual.
+#    tarball:  creates a tarball for the doc files.
+#    validate: validates
+#    clean:    removes files
+#
+# The Makefile generates an HTML version of every document.  The
+# variable DOC indicates the folder name for a given manual.
+#
+# To build a manual, you must invoke 'make' with the DOC argument.
+#
+# Examples:
+#
+#     make DOC=bitbake-user-manual
+#     make pdf DOC=bitbake-user-manual
+#
+# The first example generates the HTML version of the User Manual.
+# The second example generates the PDF version of the User Manual.
 #
 
-# You can set these variables from the command line, and also
-# from the environment for the first two.
-SPHINXOPTS    ?=
-SPHINXBUILD   ?= sphinx-build
-SOURCEDIR     = .
-BUILDDIR      = _build
-DESTDIR       = final
+ifeq ($(DOC),bitbake-user-manual)
+XSLTOPTS = --stringparam html.stylesheet bitbake-user-manual-style.css \
+           --stringparam  chapter.autolabel 1 \
+           --stringparam  section.autolabel 1 \
+           --stringparam  section.label.includes.component.label 1 \
+           --xinclude
+ALLPREQ = html tarball
+TARFILES = bitbake-user-manual-style.css bitbake-user-manual.html figures/bitbake-title.png
+MANUALS = $(DOC)/$(DOC).html
+FIGURES = figures
+STYLESHEET = $(DOC)/*.css
 
-ifeq ($(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then echo 1; else echo 0; fi),0)
-$(error "The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed")
 endif
 
-# Put it first so that "make" without argument is like "make help".
-help:
-	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+##
+# These URI should be rewritten by your distribution's xml catalog to
+# match your localy installed XSL stylesheets.
+XSL_BASE_URI  = http://docbook.sourceforge.net/release/xsl/current
+XSL_XHTML_URI = $(XSL_BASE_URI)/xhtml/docbook.xsl
 
-.PHONY: help Makefile clean publish
+all: $(ALLPREQ)
 
-publish: Makefile html singlehtml
-	rm -rf $(BUILDDIR)/$(DESTDIR)/
-	mkdir -p $(BUILDDIR)/$(DESTDIR)/
-	cp -r $(BUILDDIR)/html/* $(BUILDDIR)/$(DESTDIR)/
-	cp $(BUILDDIR)/singlehtml/index.html $(BUILDDIR)/$(DESTDIR)/singleindex.html
-	sed -i -e 's@index.html#@singleindex.html#@g' $(BUILDDIR)/$(DESTDIR)/singleindex.html
+pdf:
+ifeq ($(DOC),bitbake-user-manual)
+	@echo " "
+	@echo "********** Building."$(DOC)
+	@echo " "
+	cd $(DOC); ../tools/docbook-to-pdf $(DOC).xml ../template; cd ..
+endif
+
+html:
+ifeq ($(DOC),bitbake-user-manual)
+#       See http://www.sagehill.net/docbookxsl/HtmlOutput.html
+	@echo " "
+	@echo "******** Building "$(DOC)
+	@echo " "
+	cd $(DOC); xsltproc $(XSLTOPTS) -o $(DOC).html $(DOC)-customization.xsl $(DOC).xml; cd ..
+endif
+
+tarball: html
+	@echo " "
+	@echo "******** Creating Tarball of document files"
+	@echo " "
+	cd $(DOC); tar -cvzf $(DOC).tgz $(TARFILES); cd ..
+
+validate:
+	cd $(DOC); xmllint --postvalid --xinclude --noout $(DOC).xml; cd ..
+
+publish:
+	@if test -f $(DOC)/$(DOC).html; \
+	  then \
+            echo " "; \
+            echo "******** Publishing "$(DOC)".html"; \
+            echo " "; \
+            scp -r $(MANUALS) $(STYLESHEET) docs.yp:/var/www/www.yoctoproject.org-docs/$(VER)/$(DOC); \
+            cd $(DOC); scp -r $(FIGURES) docs.yp:/var/www/www.yoctoproject.org-docs/$(VER)/$(DOC); \
+	else \
+          echo " "; \
+          echo $(DOC)".html missing.  Generate the file first then try again."; \
+          echo " "; \
+	fi
 
 clean:
-	@rm -rf $(BUILDDIR)
-
-# Catch-all target: route all unknown targets to Sphinx using the new
-# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
-%: Makefile
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	rm -rf $(MANUALS); rm $(DOC)/$(DOC).tgz;

bitbake/doc/README

@@ -15,41 +15,25 @@ Each folder is self-contained regarding content and figures.
 If you want to find HTML versions of the BitBake manuals on the web, 
 go to http://www.openembedded.org/wiki/Documentation. 
 
-Sphinx
-======
+Makefile
+========
 
-The BitBake documentation was migrated from the original DocBook
-format to Sphinx based documentation for the Yocto Project 3.2
-release.
+The Makefile processes manual directories to create HTML, PDF,
+tarballs, etc.  Details on how the Makefile work are documented
+inside the Makefile.  See that file for more information.
 
-Additional information related to the Sphinx migration, and guidelines
-for developers willing to contribute to the BitBake documentation can
-be found in the Yocto Project Documentation README file:
+To build a manual, you run the make command and pass it the name
+of the folder containing the manual's contents. 
+For example, the following command run from the documentation directory 
+creates an HTML and a PDF version of the BitBake User Manual.
+The DOC variable specifies the manual you are making:
 
-https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/README
+     $ make DOC=bitbake-user-manual
 
-How to build the Yocto Project documentation
-============================================
+template
+========
+Contains various templates, fonts, and some old PNG files.
 
-Sphinx is written in Python. While it might work with Python2, for
-obvious reasons, we will only support building the BitBake
-documentation with Python3.
-
-Sphinx might be available in your Linux distro packages repositories,
-however it is not recommend using distro packages, as they might be
-old versions, especially if you are using an LTS version of your
-distro. The recommended method to install Sphinx and all required
-dependencies is to use the Python Package Index (pip).
-
-To install all required packages run:
-
- $ pip3 install sphinx sphinx_rtd_theme pyyaml
-
-To build the documentation locally, run:
-
- $ cd documentation
- $ make html
-
-The resulting HTML index page will be _build/html/index.html, and you
-can browse your own copy of the locally generated documentation with
-your browser.
+tools
+=====
+Contains a tool to convert the DocBook files to PDF format.

bitbake/doc/_templates/breadcrumbs.html

@@ -1,14 +0,0 @@
-{% extends "!breadcrumbs.html" %}
-
-{% block breadcrumbs %}
-    <li>
-      <span class="doctype_switcher_placeholder">{{ doctype or 'single' }}</span>
-      <span class="version_switcher_placeholder">{{ release }}</span>
-    </li>
-    <li> &raquo;</li>
-    {% for doc in parents %}
-      <li><a href="{{ doc.link|e }}">{{ doc.title }}</a> &raquo;</li>
-    {% endfor %}
-    <li>{{ title }}</li>
-{% endblock %}
-

bitbake/doc/_templates/layout.html

@@ -1,7 +0,0 @@
-{% extends "!layout.html" %}
-
-{% block extrabody %}
-<div id="outdated-warning" style="text-align: center; background-color: #FFBABA; color: #6A0E0E;">
-</div>
-{% endblock %}
-

bitbake/doc/bitbake-user-manual/bitbake-user-manual-customization.xsl

@@ -0,0 +1,29 @@
+<?xml version='1.0'?>
+<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" xmlns:fo="http://www.w3.org/1999/XSL/Format" version="1.0">
+
+  <xsl:import href="http://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-xsl-1.76.1/xhtml/docbook.xsl" />
+
+<!--
+
+  <xsl:import href="../template/1.76.1/docbook-xsl-1.76.1/xhtml/docbook.xsl" />
+
+  <xsl:import href="http://docbook.sourceforge.net/release/xsl/1.76.1/xhtml/docbook.xsl" />
+
+-->
+
+  <xsl:include href="../template/permalinks.xsl"/>
+  <xsl:include href="../template/section.title.xsl"/>
+  <xsl:include href="../template/component.title.xsl"/>
+  <xsl:include href="../template/division.title.xsl"/>
+  <xsl:include href="../template/formal.object.heading.xsl"/>
+  <xsl:include href="../template/gloss-permalinks.xsl"/>
+
+  <xsl:param name="html.stylesheet" select="'user-manual-style.css'" />
+  <xsl:param name="chapter.autolabel" select="1" />
+  <xsl:param name="section.autolabel" select="1" />
+  <xsl:param name="section.label.includes.component.label" select="1" />
+  <xsl:param name="appendix.autolabel">A</xsl:param>
+
+<!--  <xsl:param name="generate.toc" select="'article nop'"></xsl:param>  -->
+
+</xsl:stylesheet>

bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst

@@ -1,733 +0,0 @@
-.. SPDX-License-Identifier: CC-BY-2.5
-
-=========
-Execution
-=========
-
-|
-
-The primary purpose for running BitBake is to produce some kind of
-output such as a single installable package, a kernel, a software
-development kit, or even a full, board-specific bootable Linux image,
-complete with bootloader, kernel, and root filesystem. Of course, you
-can execute the ``bitbake`` command with options that cause it to
-execute single tasks, compile single recipe files, capture or clear
-data, or simply return information about the execution environment.
-
-This chapter describes BitBake's execution process from start to finish
-when you use it to create an image. The execution process is launched
-using the following command form: ::
-
-  $ bitbake target
-
-For information on
-the BitBake command and its options, see ":ref:`The BitBake Command
-<bitbake-user-manual-command>`" section.
-
-.. note::
-
-   Prior to executing BitBake, you should take advantage of available
-   parallel thread execution on your build host by setting the
-   :term:`BB_NUMBER_THREADS` variable in
-   your project's ``local.conf`` configuration file.
-
-   A common method to determine this value for your build host is to run
-   the following: ::
-
-     $ grep processor /proc/cpuinfo
-
-   This command returns
-   the number of processors, which takes into account hyper-threading.
-   Thus, a quad-core build host with hyper-threading most likely shows
-   eight processors, which is the value you would then assign to
-   ``BB_NUMBER_THREADS``.
-
-   A possibly simpler solution is that some Linux distributions (e.g.
-   Debian and Ubuntu) provide the ``ncpus`` command.
-
-Parsing the Base Configuration Metadata
-=======================================
-
-The first thing BitBake does is parse base configuration metadata. Base
-configuration metadata consists of your project's ``bblayers.conf`` file
-to determine what layers BitBake needs to recognize, all necessary
-``layer.conf`` files (one from each layer), and ``bitbake.conf``. The
-data itself is of various types:
-
--  **Recipes:** Details about particular pieces of software.
-
--  **Class Data:** An abstraction of common build information (e.g. how to
-   build a Linux kernel).
-
--  **Configuration Data:** Machine-specific settings, policy decisions,
-   and so forth. Configuration data acts as the glue to bind everything
-   together.
-
-The ``layer.conf`` files are used to construct key variables such as
-:term:`BBPATH` and :term:`BBFILES`.
-``BBPATH`` is used to search for configuration and class files under the
-``conf`` and ``classes`` directories, respectively. ``BBFILES`` is used
-to locate both recipe and recipe append files (``.bb`` and
-``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the
-user has set the ``BBPATH`` and ``BBFILES`` directly in the environment.
-
-Next, the ``bitbake.conf`` file is located using the ``BBPATH`` variable
-that was just constructed. The ``bitbake.conf`` file may also include
-other configuration files using the ``include`` or ``require``
-directives.
-
-Prior to parsing configuration files, BitBake looks at certain
-variables, including:
-
--  :term:`BB_ENV_WHITELIST`
--  :term:`BB_ENV_EXTRAWHITE`
--  :term:`BB_PRESERVE_ENV`
--  :term:`BB_ORIGENV`
--  :term:`BITBAKE_UI`
-
-The first four variables in this list relate to how BitBake treats shell
-environment variables during task execution. By default, BitBake cleans
-the environment variables and provides tight control over the shell
-execution environment. However, through the use of these first four
-variables, you can apply your control regarding the environment
-variables allowed to be used by BitBake in the shell during execution of
-tasks. See the
-":ref:`bitbake-user-manual/bitbake-user-manual-metadata:Passing Information Into the Build Task Environment`"
-section and the information about these variables in the variable
-glossary for more information on how they work and on how to use them.
-
-The base configuration metadata is global and therefore affects all
-recipes and tasks that are executed.
-
-BitBake first searches the current working directory for an optional
-``conf/bblayers.conf`` configuration file. This file is expected to
-contain a :term:`BBLAYERS` variable that is a
-space-delimited list of 'layer' directories. Recall that if BitBake
-cannot find a ``bblayers.conf`` file, then it is assumed the user has
-set the ``BBPATH`` and ``BBFILES`` variables directly in the
-environment.
-
-For each directory (layer) in this list, a ``conf/layer.conf`` file is
-located and parsed with the :term:`LAYERDIR` variable
-being set to the directory where the layer was found. The idea is these
-files automatically set up :term:`BBPATH` and other
-variables correctly for a given build directory.
-
-BitBake then expects to find the ``conf/bitbake.conf`` file somewhere in
-the user-specified ``BBPATH``. That configuration file generally has
-include directives to pull in any other metadata such as files specific
-to the architecture, the machine, the local environment, and so forth.
-
-Only variable definitions and include directives are allowed in BitBake
-``.conf`` files. Some variables directly influence BitBake's behavior.
-These variables might have been set from the environment depending on
-the environment variables previously mentioned or set in the
-configuration files. The ":ref:`bitbake-user-manual/bitbake-user-manual-ref-variables:Variables Glossary`"
-chapter presents a full list of
-variables.
-
-After parsing configuration files, BitBake uses its rudimentary
-inheritance mechanism, which is through class files, to inherit some
-standard classes. BitBake parses a class when the inherit directive
-responsible for getting that class is encountered.
-
-The ``base.bbclass`` file is always included. Other classes that are
-specified in the configuration using the
-:term:`INHERIT` variable are also included. BitBake
-searches for class files in a ``classes`` subdirectory under the paths
-in ``BBPATH`` in the same way as configuration files.
-
-A good way to get an idea of the configuration files and the class files
-used in your execution environment is to run the following BitBake
-command: ::
-
-  $ bitbake -e > mybb.log
-
-Examining the top of the ``mybb.log``
-shows you the many configuration files and class files used in your
-execution environment.
-
-.. note::
-
-   You need to be aware of how BitBake parses curly braces. If a recipe
-   uses a closing curly brace within the function and the character has
-   no leading spaces, BitBake produces a parsing error. If you use a
-   pair of curly braces in a shell function, the closing curly brace
-   must not be located at the start of the line without leading spaces.
-
-   Here is an example that causes BitBake to produce a parsing error: ::
-
-      fakeroot create_shar() {
-         cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
-      usage()
-      {
-         echo "test"
-         ######  The following "}" at the start of the line causes a parsing error ######
-      }
-      EOF
-      }
-
-      Writing the recipe this way avoids the error:
-      fakeroot create_shar() {
-         cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
-      usage()
-      {
-         echo "test"
-         ###### The following "}" with a leading space at the start of the line avoids the error ######
-       }
-      EOF
-      }
-
-Locating and Parsing Recipes
-============================
-
-During the configuration phase, BitBake will have set
-:term:`BBFILES`. BitBake now uses it to construct a
-list of recipes to parse, along with any append files (``.bbappend``) to
-apply. ``BBFILES`` is a space-separated list of available files and
-supports wildcards. An example would be: ::
-
-  BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend"
-
-BitBake parses each
-recipe and append file located with ``BBFILES`` and stores the values of
-various variables into the datastore.
-
-.. note::
-
-   Append files are applied in the order they are encountered in BBFILES.
-
-For each file, a fresh copy of the base configuration is made, then the
-recipe is parsed line by line. Any inherit statements cause BitBake to
-find and then parse class files (``.bbclass``) using
-:term:`BBPATH` as the search path. Finally, BitBake
-parses in order any append files found in ``BBFILES``.
-
-One common convention is to use the recipe filename to define pieces of
-metadata. For example, in ``bitbake.conf`` the recipe name and version
-are used to set the variables :term:`PN` and
-:term:`PV`: ::
-
-   PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
-   PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
-
-In this example, a recipe called "something_1.2.3.bb" would set
-``PN`` to "something" and ``PV`` to "1.2.3".
-
-By the time parsing is complete for a recipe, BitBake has a list of
-tasks that the recipe defines and a set of data consisting of keys and
-values as well as dependency information about the tasks.
-
-BitBake does not need all of this information. It only needs a small
-subset of the information to make decisions about the recipe.
-Consequently, BitBake caches the values in which it is interested and
-does not store the rest of the information. Experience has shown it is
-faster to re-parse the metadata than to try and write it out to the disk
-and then reload it.
-
-Where possible, subsequent BitBake commands reuse this cache of recipe
-information. The validity of this cache is determined by first computing
-a checksum of the base configuration data (see
-:term:`BB_HASHCONFIG_WHITELIST`) and
-then checking if the checksum matches. If that checksum matches what is
-in the cache and the recipe and class files have not changed, BitBake is
-able to use the cache. BitBake then reloads the cached information about
-the recipe instead of reparsing it from scratch.
-
-Recipe file collections exist to allow the user to have multiple
-repositories of ``.bb`` files that contain the same exact package. For
-example, one could easily use them to make one's own local copy of an
-upstream repository, but with custom modifications that one does not
-want upstream. Here is an example: ::
-
-  BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb"
-  BBFILE_COLLECTIONS = "upstream local"
-  BBFILE_PATTERN_upstream = "^/stuff/openembedded/"
-  BBFILE_PATTERN_local = "^/stuff/openembedded.modified/"
-  BBFILE_PRIORITY_upstream = "5" BBFILE_PRIORITY_local = "10"
-
-.. note::
-
-   The layers mechanism is now the preferred method of collecting code.
-   While the collections code remains, its main use is to set layer
-   priorities and to deal with overlap (conflicts) between layers.
-
-.. _bb-bitbake-providers:
-
-Providers
-=========
-
-Assuming BitBake has been instructed to execute a target and that all
-the recipe files have been parsed, BitBake starts to figure out how to
-build the target. BitBake looks through the ``PROVIDES`` list for each
-of the recipes. A ``PROVIDES`` list is the list of names by which the
-recipe can be known. Each recipe's ``PROVIDES`` list is created
-implicitly through the recipe's :term:`PN` variable and
-explicitly through the recipe's :term:`PROVIDES`
-variable, which is optional.
-
-When a recipe uses ``PROVIDES``, that recipe's functionality can be
-found under an alternative name or names other than the implicit ``PN``
-name. As an example, suppose a recipe named ``keyboard_1.0.bb``
-contained the following: ::
-
-  PROVIDES += "fullkeyboard"
-
-The ``PROVIDES``
-list for this recipe becomes "keyboard", which is implicit, and
-"fullkeyboard", which is explicit. Consequently, the functionality found
-in ``keyboard_1.0.bb`` can be found under two different names.
-
-.. _bb-bitbake-preferences:
-
-Preferences
-===========
-
-The ``PROVIDES`` list is only part of the solution for figuring out a
-target's recipes. Because targets might have multiple providers, BitBake
-needs to prioritize providers by determining provider preferences.
-
-A common example in which a target has multiple providers is
-"virtual/kernel", which is on the ``PROVIDES`` list for each kernel
-recipe. Each machine often selects the best kernel provider by using a
-line similar to the following in the machine configuration file: ::
-
-  PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
-
-The default :term:`PREFERRED_PROVIDER` is the provider
-with the same name as the target. BitBake iterates through each target
-it needs to build and resolves them and their dependencies using this
-process.
-
-Understanding how providers are chosen is made complicated by the fact
-that multiple versions might exist for a given provider. BitBake
-defaults to the highest version of a provider. Version comparisons are
-made using the same method as Debian. You can use the
-:term:`PREFERRED_VERSION` variable to
-specify a particular version. You can influence the order by using the
-:term:`DEFAULT_PREFERENCE` variable.
-
-By default, files have a preference of "0". Setting
-``DEFAULT_PREFERENCE`` to "-1" makes the recipe unlikely to be used
-unless it is explicitly referenced. Setting ``DEFAULT_PREFERENCE`` to
-"1" makes it likely the recipe is used. ``PREFERRED_VERSION`` overrides
-any ``DEFAULT_PREFERENCE`` setting. ``DEFAULT_PREFERENCE`` is often used
-to mark newer and more experimental recipe versions until they have
-undergone sufficient testing to be considered stable.
-
-When there are multiple "versions" of a given recipe, BitBake defaults
-to selecting the most recent version, unless otherwise specified. If the
-recipe in question has a
-:term:`DEFAULT_PREFERENCE` set lower than
-the other recipes (default is 0), then it will not be selected. This
-allows the person or persons maintaining the repository of recipe files
-to specify their preference for the default selected version.
-Additionally, the user can specify their preferred version.
-
-If the first recipe is named ``a_1.1.bb``, then the
-:term:`PN` variable will be set to "a", and the
-:term:`PV` variable will be set to 1.1.
-
-Thus, if a recipe named ``a_1.2.bb`` exists, BitBake will choose 1.2 by
-default. However, if you define the following variable in a ``.conf``
-file that BitBake parses, you can change that preference: ::
-
-  PREFERRED_VERSION_a = "1.1"
-
-.. note::
-
-   It is common for a recipe to provide two versions -- a stable,
-   numbered (and preferred) version, and a version that is automatically
-   checked out from a source code repository that is considered more
-   "bleeding edge" but can be selected only explicitly.
-
-   For example, in the OpenEmbedded codebase, there is a standard,
-   versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there
-   is also a Git-based version, ``busybox_git.bb``, which explicitly
-   contains the line ::
-
-     DEFAULT_PREFERENCE = "-1"
-
-   to ensure that the
-   numbered, stable version is always preferred unless the developer
-   selects otherwise.
-
-.. _bb-bitbake-dependencies:
-
-Dependencies
-============
-
-Each target BitBake builds consists of multiple tasks such as ``fetch``,
-``unpack``, ``patch``, ``configure``, and ``compile``. For best
-performance on multi-core systems, BitBake considers each task as an
-independent entity with its own set of dependencies.
-
-Dependencies are defined through several variables. You can find
-information about variables BitBake uses in the
-:doc:`bitbake-user-manual-ref-variables` near the end of this manual. At a
-basic level, it is sufficient to know that BitBake uses the
-:term:`DEPENDS` and
-:term:`RDEPENDS` variables when calculating
-dependencies.
-
-For more information on how BitBake handles dependencies, see the
-:ref:`bitbake-user-manual/bitbake-user-manual-metadata:Dependencies`
-section.
-
-.. _ref-bitbake-tasklist:
-
-The Task List
-=============
-
-Based on the generated list of providers and the dependency information,
-BitBake can now calculate exactly what tasks it needs to run and in what
-order it needs to run them. The
-:ref:`bitbake-user-manual/bitbake-user-manual-execution:executing tasks`
-section has more information on how BitBake chooses which task to
-execute next.
-
-The build now starts with BitBake forking off threads up to the limit
-set in the :term:`BB_NUMBER_THREADS`
-variable. BitBake continues to fork threads as long as there are tasks
-ready to run, those tasks have all their dependencies met, and the
-thread threshold has not been exceeded.
-
-It is worth noting that you can greatly speed up the build time by
-properly setting the ``BB_NUMBER_THREADS`` variable.
-
-As each task completes, a timestamp is written to the directory
-specified by the :term:`STAMP` variable. On subsequent
-runs, BitBake looks in the build directory within ``tmp/stamps`` and
-does not rerun tasks that are already completed unless a timestamp is
-found to be invalid. Currently, invalid timestamps are only considered
-on a per recipe file basis. So, for example, if the configure stamp has
-a timestamp greater than the compile timestamp for a given target, then
-the compile task would rerun. Running the compile task again, however,
-has no effect on other providers that depend on that target.
-
-The exact format of the stamps is partly configurable. In modern
-versions of BitBake, a hash is appended to the stamp so that if the
-configuration changes, the stamp becomes invalid and the task is
-automatically rerun. This hash, or signature used, is governed by the
-signature policy that is configured (see the
-:ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`
-section for information). It is also
-possible to append extra metadata to the stamp using the
-``[stamp-extra-info]`` task flag. For example, OpenEmbedded uses this
-flag to make some tasks machine-specific.
-
-.. note::
-
-   Some tasks are marked as "nostamp" tasks. No timestamp file is
-   created when these tasks are run. Consequently, "nostamp" tasks are
-   always rerun.
-
-For more information on tasks, see the
-:ref:`bitbake-user-manual/bitbake-user-manual-metadata:tasks` section.
-
-Executing Tasks
-===============
-
-Tasks can be either a shell task or a Python task. For shell tasks,
-BitBake writes a shell script to
-``${``\ :term:`T`\ ``}/run.do_taskname.pid`` and then
-executes the script. The generated shell script contains all the
-exported variables, and the shell functions with all variables expanded.
-Output from the shell script goes to the file
-``${T}/log.do_taskname.pid``. Looking at the expanded shell functions in
-the run file and the output in the log files is a useful debugging
-technique.
-
-For Python tasks, BitBake executes the task internally and logs
-information to the controlling terminal. Future versions of BitBake will
-write the functions to files similar to the way shell tasks are handled.
-Logging will be handled in a way similar to shell tasks as well.
-
-The order in which BitBake runs the tasks is controlled by its task
-scheduler. It is possible to configure the scheduler and define custom
-implementations for specific use cases. For more information, see these
-variables that control the behavior:
-
--  :term:`BB_SCHEDULER`
-
--  :term:`BB_SCHEDULERS`
-
-It is possible to have functions run before and after a task's main
-function. This is done using the ``[prefuncs]`` and ``[postfuncs]``
-flags of the task that lists the functions to run.
-
-.. _checksums:
-
-Checksums (Signatures)
-======================
-
-A checksum is a unique signature of a task's inputs. The signature of a
-task can be used to determine if a task needs to be run. Because it is a
-change in a task's inputs that triggers running the task, BitBake needs
-to detect all the inputs to a given task. For shell tasks, this turns
-out to be fairly easy because BitBake generates a "run" shell script for
-each task and it is possible to create a checksum that gives you a good
-idea of when the task's data changes.
-
-To complicate the problem, some things should not be included in the
-checksum. First, there is the actual specific build path of a given task
-- the working directory. It does not matter if the working directory
-changes because it should not affect the output for target packages. The
-simplistic approach for excluding the working directory is to set it to
-some fixed value and create the checksum for the "run" script. BitBake
-goes one step better and uses the
-:term:`BB_HASHBASE_WHITELIST` variable
-to define a list of variables that should never be included when
-generating the signatures.
-
-Another problem results from the "run" scripts containing functions that
-might or might not get called. The incremental build solution contains
-code that figures out dependencies between shell functions. This code is
-used to prune the "run" scripts down to the minimum set, thereby
-alleviating this problem and making the "run" scripts much more readable
-as a bonus.
-
-So far we have solutions for shell scripts. What about Python tasks? The
-same approach applies even though these tasks are more difficult. The
-process needs to figure out what variables a Python function accesses
-and what functions it calls. Again, the incremental build solution
-contains code that first figures out the variable and function
-dependencies, and then creates a checksum for the data used as the input
-to the task.
-
-Like the working directory case, situations exist where dependencies
-should be ignored. For these cases, you can instruct the build process
-to ignore a dependency by using a line like the following: ::
-
-  PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
-
-This example ensures that the
-``PACKAGE_ARCHS`` variable does not depend on the value of ``MACHINE``,
-even if it does reference it.
-
-Equally, there are cases where we need to add dependencies BitBake is
-not able to find. You can accomplish this by using a line like the
-following: ::
-
-  PACKAGE_ARCHS[vardeps] = "MACHINE"
-
-This example explicitly
-adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``.
-
-Consider a case with in-line Python, for example, where BitBake is not
-able to figure out dependencies. When running in debug mode (i.e. using
-``-DDD``), BitBake produces output when it discovers something for which
-it cannot figure out dependencies.
-
-Thus far, this section has limited discussion to the direct inputs into
-a task. Information based on direct inputs is referred to as the
-"basehash" in the code. However, there is still the question of a task's
-indirect inputs - the things that were already built and present in the
-build directory. The checksum (or signature) for a particular task needs
-to add the hashes of all the tasks on which the particular task depends.
-Choosing which dependencies to add is a policy decision. However, the
-effect is to generate a master checksum that combines the basehash and
-the hashes of the task's dependencies.
-
-At the code level, there are a variety of ways both the basehash and the
-dependent task hashes can be influenced. Within the BitBake
-configuration file, we can give BitBake some extra information to help
-it construct the basehash. The following statement effectively results
-in a list of global variable dependency excludes - variables never
-included in any checksum. This example uses variables from OpenEmbedded
-to help illustrate the concept: ::
-
-   BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
-       SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
-       USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
-       PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
-       CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
-
-The previous example excludes the work directory, which is part of
-``TMPDIR``.
-
-The rules for deciding which hashes of dependent tasks to include
-through dependency chains are more complex and are generally
-accomplished with a Python function. The code in
-``meta/lib/oe/sstatesig.py`` shows two examples of this and also
-illustrates how you can insert your own policy into the system if so
-desired. This file defines the two basic signature generators
-OpenEmbedded-Core uses: "OEBasic" and "OEBasicHash". By default, there
-is a dummy "noop" signature handler enabled in BitBake. This means that
-behavior is unchanged from previous versions. ``OE-Core`` uses the
-"OEBasicHash" signature handler by default through this setting in the
-``bitbake.conf`` file: ::
-
-  BB_SIGNATURE_HANDLER ?= "OEBasicHash"
-
-The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same as the "OEBasic"
-version but adds the task hash to the stamp files. This results in any
-metadata change that changes the task hash, automatically causing the
-task to be run again. This removes the need to bump
-:term:`PR` values, and changes to metadata automatically
-ripple across the build.
-
-It is also worth noting that the end result of these signature
-generators is to make some dependency and hash information available to
-the build. This information includes:
-
--  ``BB_BASEHASH_task-``\ *taskname*: The base hashes for each task in the
-   recipe.
-
--  ``BB_BASEHASH_``\ *filename:taskname*: The base hashes for each
-   dependent task.
-
--  ``BBHASHDEPS_``\ *filename:taskname*: The task dependencies for
-   each task.
-
--  ``BB_TASKHASH``: The hash of the currently running task.
-
-It is worth noting that BitBake's "-S" option lets you debug BitBake's
-processing of signatures. The options passed to -S allow different
-debugging modes to be used, either using BitBake's own debug functions
-or possibly those defined in the metadata/signature handler itself. The
-simplest parameter to pass is "none", which causes a set of signature
-information to be written out into ``STAMPS_DIR`` corresponding to the
-targets specified. The other currently available parameter is
-"printdiff", which causes BitBake to try to establish the closest
-signature match it can (e.g. in the sstate cache) and then run
-``bitbake-diffsigs`` over the matches to determine the stamps and delta
-where these two stamp trees diverge.
-
-.. note::
-
-   It is likely that future versions of BitBake will provide other
-   signature handlers triggered through additional "-S" parameters.
-
-You can find more information on checksum metadata in the
-:ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene`
-section.
-
-Setscene
-========
-
-The setscene process enables BitBake to handle "pre-built" artifacts.
-The ability to handle and reuse these artifacts allows BitBake the
-luxury of not having to build something from scratch every time.
-Instead, BitBake can use, when possible, existing build artifacts.
-
-BitBake needs to have reliable data indicating whether or not an
-artifact is compatible. Signatures, described in the previous section,
-provide an ideal way of representing whether an artifact is compatible.
-If a signature is the same, an object can be reused.
-
-If an object can be reused, the problem then becomes how to replace a
-given task or set of tasks with the pre-built artifact. BitBake solves
-the problem with the "setscene" process.
-
-When BitBake is asked to build a given target, before building anything,
-it first asks whether cached information is available for any of the
-targets it's building, or any of the intermediate targets. If cached
-information is available, BitBake uses this information instead of
-running the main tasks.
-
-BitBake first calls the function defined by the
-:term:`BB_HASHCHECK_FUNCTION` variable
-with a list of tasks and corresponding hashes it wants to build. This
-function is designed to be fast and returns a list of the tasks for
-which it believes in can obtain artifacts.
-
-Next, for each of the tasks that were returned as possibilities, BitBake
-executes a setscene version of the task that the possible artifact
-covers. Setscene versions of a task have the string "_setscene" appended
-to the task name. So, for example, the task with the name ``xxx`` has a
-setscene task named ``xxx_setscene``. The setscene version of the task
-executes and provides the necessary artifacts returning either success
-or failure.
-
-As previously mentioned, an artifact can cover more than one task. For
-example, it is pointless to obtain a compiler if you already have the
-compiled binary. To handle this, BitBake calls the
-:term:`BB_SETSCENE_DEPVALID` function for
-each successful setscene task to know whether or not it needs to obtain
-the dependencies of that task.
-
-Finally, after all the setscene tasks have executed, BitBake calls the
-function listed in
-:term:`BB_SETSCENE_VERIFY_FUNCTION2`
-with the list of tasks BitBake thinks has been "covered". The metadata
-can then ensure that this list is correct and can inform BitBake that it
-wants specific tasks to be run regardless of the setscene result.
-
-You can find more information on setscene metadata in the
-:ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene`
-section.
-
-Logging
-=======
-
-In addition to the standard command line option to control how verbose
-builds are when execute, bitbake also supports user defined
-configuration of the `Python
-logging <https://docs.python.org/3/library/logging.html>`__ facilities
-through the :term:`BB_LOGCONFIG` variable. This
-variable defines a json or yaml `logging
-configuration <https://docs.python.org/3/library/logging.config.html>`__
-that will be intelligently merged into the default configuration. The
-logging configuration is merged using the following rules:
-
--  The user defined configuration will completely replace the default
-   configuration if top level key ``bitbake_merge`` is set to the value
-   ``False``. In this case, all other rules are ignored.
-
--  The user configuration must have a top level ``version`` which must
-   match the value of the default configuration.
-
--  Any keys defined in the ``handlers``, ``formatters``, or ``filters``,
-   will be merged into the same section in the default configuration,
-   with the user specified keys taking replacing a default one if there
-   is a conflict. In practice, this means that if both the default
-   configuration and user configuration specify a handler named
-   ``myhandler``, the user defined one will replace the default. To
-   prevent the user from inadvertently replacing a default handler,
-   formatter, or filter, all of the default ones are named with a prefix
-   of "``BitBake.``"
-
--  If a logger is defined by the user with the key ``bitbake_merge`` set
-   to ``False``, that logger will be completely replaced by user
-   configuration. In this case, no other rules will apply to that
-   logger.
-
--  All user defined ``filter`` and ``handlers`` properties for a given
-   logger will be merged with corresponding properties from the default
-   logger. For example, if the user configuration adds a filter called
-   ``myFilter`` to the ``BitBake.SigGen``, and the default configuration
-   adds a filter called ``BitBake.defaultFilter``, both filters will be
-   applied to the logger
-
-As an example, consider the following user logging configuration file
-which logs all Hash Equivalence related messages of VERBOSE or higher to
-a file called ``hashequiv.log`` ::
-
-   {
-       "version": 1,
-       "handlers": {
-           "autobuilderlog": {
-               "class": "logging.FileHandler",
-               "formatter": "logfileFormatter",
-               "level": "DEBUG",
-               "filename": "hashequiv.log",
-               "mode": "w"
-           }
-       },
-       "formatters": {
-               "logfileFormatter": {
-                   "format": "%(name)s: %(levelname)s: %(message)s"
-               }
-       },
-       "loggers": {
-           "BitBake.SigGen.HashEquiv": {
-               "level": "VERBOSE",
-               "handlers": ["autobuilderlog"]
-           },
-           "BitBake.RunQueue.HashEquiv": {
-               "level": "VERBOSE",
-               "handlers": ["autobuilderlog"]
-           }
-       }
-   }

bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml

@@ -0,0 +1,932 @@
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+
+<chapter id="bitbake-user-manual-execution">
+    <title>Execution</title>
+
+    <para>
+        The primary purpose for running BitBake is to produce some kind
+        of output such as a single installable package, a kernel, a software
+        development kit, or even a full, board-specific bootable Linux image,
+        complete with bootloader, kernel, and root filesystem.
+        Of course, you can execute the <filename>bitbake</filename>
+        command with options that cause it to execute single tasks,
+        compile single recipe files, capture or clear data, or simply
+        return information about the execution environment.
+    </para>
+
+    <para>
+        This chapter describes BitBake's execution process from start
+        to finish when you use it to create an image.
+        The execution process is launched using the following command
+        form:
+        <literallayout class='monospaced'>
+     $ bitbake <replaceable>target</replaceable>
+        </literallayout>
+        For information on the BitBake command and its options,
+        see
+        "<link linkend='bitbake-user-manual-command'>The BitBake Command</link>"
+        section.
+        <note>
+            <para>
+                Prior to executing BitBake, you should take advantage of available
+                parallel thread execution on your build host by setting the
+                <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
+                variable in your project's <filename>local.conf</filename>
+                configuration file.
+            </para>
+
+            <para>
+                A common method to determine this value for your build host is to run
+                the following:
+                <literallayout class='monospaced'>
+     $ grep processor /proc/cpuinfo
+                </literallayout>
+                This command returns the number of processors, which takes into
+                account hyper-threading.
+                Thus, a quad-core build host with hyper-threading most likely
+                shows eight processors, which is the value you would then assign to
+                <filename>BB_NUMBER_THREADS</filename>.
+            </para>
+
+            <para>
+                A possibly simpler solution is that some Linux distributions
+                (e.g. Debian and Ubuntu) provide the <filename>ncpus</filename> command.
+            </para>
+        </note>
+    </para>
+
+    <section id='parsing-the-base-configuration-metadata'>
+        <title>Parsing the Base Configuration Metadata</title>
+
+        <para>
+            The first thing BitBake does is parse base configuration
+            metadata.
+            Base configuration metadata consists of your project's
+            <filename>bblayers.conf</filename> file to determine what
+            layers BitBake needs to recognize, all necessary
+            <filename>layer.conf</filename> files (one from each layer),
+            and <filename>bitbake.conf</filename>.
+            The data itself is of various types:
+            <itemizedlist>
+                <listitem><para><emphasis>Recipes:</emphasis>
+                    Details about particular pieces of software.
+                    </para></listitem>
+                <listitem><para><emphasis>Class Data:</emphasis>
+                    An abstraction of common build information
+                    (e.g. how to build a Linux kernel).
+                    </para></listitem>
+                <listitem><para><emphasis>Configuration Data:</emphasis>
+                    Machine-specific settings, policy decisions,
+                    and so forth.
+                    Configuration data acts as the glue to bind everything
+                    together.</para></listitem>
+            </itemizedlist>
+        </para>
+
+        <para>
+            The <filename>layer.conf</filename> files are used to
+            construct key variables such as
+            <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
+            and
+            <link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
+            <filename>BBPATH</filename> is used to search for
+            configuration and class files under the
+            <filename>conf</filename> and <filename>classes</filename>
+            directories, respectively.
+            <filename>BBFILES</filename> is used to locate both recipe
+            and recipe append files
+            (<filename>.bb</filename> and <filename>.bbappend</filename>).
+            If there is no <filename>bblayers.conf</filename> file,
+            it is assumed the user has set the <filename>BBPATH</filename>
+            and <filename>BBFILES</filename> directly in the environment.
+        </para>
+
+        <para>
+            Next, the <filename>bitbake.conf</filename> file is located
+            using the <filename>BBPATH</filename> variable that was
+            just constructed.
+            The <filename>bitbake.conf</filename> file may also include other
+            configuration files using the
+            <filename>include</filename> or
+            <filename>require</filename> directives.
+        </para>
+
+        <para>
+            Prior to parsing configuration files, Bitbake looks
+            at certain variables, including:
+            <itemizedlist>
+                <listitem><para>
+                    <link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
+                    </para></listitem>
+                <listitem><para>
+                    <link linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
+                    </para></listitem>
+                <listitem><para>
+                    <link linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
+                    </para></listitem>
+                <listitem><para>
+                    <link linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
+                    </para></listitem>
+                <listitem><para>
+                    <link linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
+                    </para></listitem>
+            </itemizedlist>
+            The first four variables in this list relate to how BitBake treats shell
+            environment variables during task execution.
+            By default, BitBake cleans the environment variables and provides tight
+            control over the shell execution environment.
+            However, through the use of these first four variables, you can
+            apply your control regarding the
+            environment variables allowed to be used by BitBake in the shell
+            during execution of tasks.
+            See the
+            "<link linkend='passing-information-into-the-build-task-environment'>Passing Information Into the Build Task Environment</link>"
+            section and the information about these variables in the
+            variable glossary for more information on how they work and
+            on how to use them.
+        </para>
+
+        <para>
+            The base configuration metadata is global
+            and therefore affects all recipes and tasks that are executed.
+        </para>
+
+        <para>
+            BitBake first searches the current working directory for an
+            optional <filename>conf/bblayers.conf</filename> configuration file.
+            This file is expected to contain a
+            <link linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link>
+            variable that is a space-delimited list of 'layer' directories.
+            Recall that if BitBake cannot find a <filename>bblayers.conf</filename>
+            file, then it is assumed the user has set the <filename>BBPATH</filename>
+            and <filename>BBFILES</filename> variables directly in the environment.
+        </para>
+
+        <para>
+            For each directory (layer) in this list, a <filename>conf/layer.conf</filename>
+            file is located and parsed with the
+            <link linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link>
+            variable being set to the directory where the layer was found.
+            The idea is these files automatically set up
+            <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
+            and other variables correctly for a given build directory.
+        </para>
+
+        <para>
+            BitBake then expects to find the <filename>conf/bitbake.conf</filename>
+            file somewhere in the user-specified <filename>BBPATH</filename>.
+            That configuration file generally has include directives to pull
+            in any other metadata such as files specific to the architecture,
+            the machine, the local environment, and so forth.
+        </para>
+
+        <para>
+            Only variable definitions and include directives are allowed
+            in BitBake <filename>.conf</filename> files.
+            Some variables directly influence BitBake's behavior.
+            These variables might have been set from the environment
+            depending on the environment variables previously
+            mentioned or set in the configuration files.
+            The
+            "<link linkend='ref-variables-glos'>Variables Glossary</link>"
+            chapter presents a full list of variables.
+        </para>
+
+        <para>
+            After parsing configuration files, BitBake uses its rudimentary
+            inheritance mechanism, which is through class files, to inherit
+            some standard classes.
+            BitBake parses a class when the inherit directive responsible
+            for getting that class is encountered.
+        </para>
+
+        <para>
+            The <filename>base.bbclass</filename> file is always included.
+            Other classes that are specified in the configuration using the
+            <link linkend='var-INHERIT'><filename>INHERIT</filename></link>
+            variable are also included.
+            BitBake searches for class files in a
+            <filename>classes</filename> subdirectory under
+            the paths in <filename>BBPATH</filename> in the same way as
+            configuration files.
+        </para>
+
+        <para>
+            A good way to get an idea of the configuration files and
+            the class files used in your execution environment is to
+            run the following BitBake command:
+            <literallayout class='monospaced'>
+     $ bitbake -e > mybb.log
+            </literallayout>
+            Examining the top of the <filename>mybb.log</filename>
+            shows you the many configuration files and class files
+            used in your execution environment.
+        </para>
+
+        <note>
+            <para>
+                You need to be aware of how BitBake parses curly braces.
+                If a recipe uses a closing curly brace within the function and
+                the character has no leading spaces, BitBake produces a parsing
+                error.
+                If you use a pair of curly braces in a shell function, the
+                closing curly brace must not be located at the start of the line
+                without leading spaces.
+            </para>
+
+            <para>
+                Here is an example that causes BitBake to produce a parsing
+                error:
+                <literallayout class='monospaced'>
+     fakeroot create_shar() {
+         cat &lt;&lt; "EOF" &gt; ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+     usage()
+     {
+       echo "test"
+       ###### The following "}" at the start of the line causes a parsing error ######
+     }
+     EOF
+     }
+                </literallayout>
+                Writing the recipe this way avoids the error:
+                <literallayout class='monospaced'>
+     fakeroot create_shar() {
+         cat &lt;&lt; "EOF" &gt; ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+     usage()
+     {
+       echo "test"
+       ######The following "}" with a leading space at the start of the line avoids the error ######
+      }
+     EOF
+     }
+                </literallayout>
+            </para>
+        </note>
+    </section>
+
+    <section id='locating-and-parsing-recipes'>
+        <title>Locating and Parsing Recipes</title>
+
+        <para>
+            During the configuration phase, BitBake will have set
+            <link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
+            BitBake now uses it to construct a list of recipes to parse,
+            along with any append files (<filename>.bbappend</filename>)
+            to apply.
+            <filename>BBFILES</filename> is a space-separated list of
+            available files and supports wildcards.
+            An example would be:
+            <literallayout class='monospaced'>
+     BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend"
+            </literallayout>
+            BitBake parses each recipe and append file located
+            with <filename>BBFILES</filename> and stores the values of
+            various variables into the datastore.
+            <note>
+                Append files are applied in the order they are encountered in
+                <filename>BBFILES</filename>.
+            </note>
+            For each file, a fresh copy of the base configuration is
+            made, then the recipe is parsed line by line.
+            Any inherit statements cause BitBake to find and
+            then parse class files (<filename>.bbclass</filename>)
+            using
+            <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
+            as the search path.
+            Finally, BitBake parses in order any append files found in
+            <filename>BBFILES</filename>.
+        </para>
+
+        <para>
+            One common convention is to use the recipe filename to define
+            pieces of metadata.
+            For example, in <filename>bitbake.conf</filename> the recipe
+            name and version are used to set the variables
+            <link linkend='var-PN'><filename>PN</filename></link> and
+            <link linkend='var-PV'><filename>PV</filename></link>:
+            <literallayout class='monospaced'>
+     PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
+     PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
+            </literallayout>
+            In this example, a recipe called "something_1.2.3.bb" would set
+            <filename>PN</filename> to "something" and
+            <filename>PV</filename> to "1.2.3".
+        </para>
+
+        <para>
+            By the time parsing is complete for a recipe, BitBake
+            has a list of tasks that the recipe defines and a set of
+            data consisting of keys and values as well as
+            dependency information about the tasks.
+        </para>
+
+        <para>
+            BitBake does not need all of this information.
+            It only needs a small subset of the information to make
+            decisions about the recipe.
+            Consequently, BitBake caches the values in which it is
+            interested and does not store the rest of the information.
+            Experience has shown it is faster to re-parse the metadata than to
+            try and write it out to the disk and then reload it.
+        </para>
+
+        <para>
+            Where possible, subsequent BitBake commands reuse this cache of
+            recipe information.
+            The validity of this cache is determined by first computing a
+            checksum of the base configuration data (see
+            <link linkend='var-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
+            and then checking if the checksum matches.
+            If that checksum matches what is in the cache and the recipe
+            and class files have not changed, Bitbake is able to use
+            the cache.
+            BitBake then reloads the cached information about the recipe
+            instead of reparsing it from scratch.
+        </para>
+
+        <para>
+            Recipe file collections exist to allow the user to
+            have multiple repositories of
+            <filename>.bb</filename> files that contain the same
+            exact package.
+            For example, one could easily use them to make one's
+            own local copy of an upstream repository, but with
+            custom modifications that one does not want upstream.
+            Here is an example:
+            <literallayout class='monospaced'>
+    BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb"
+    BBFILE_COLLECTIONS = "upstream local"
+    BBFILE_PATTERN_upstream = "^/stuff/openembedded/"
+    BBFILE_PATTERN_local = "^/stuff/openembedded.modified/"
+    BBFILE_PRIORITY_upstream = "5"
+    BBFILE_PRIORITY_local = "10"
+            </literallayout>
+            <note>
+                The layers mechanism is now the preferred method of collecting
+                code.
+                While the collections code remains, its main use is to set layer
+                priorities and to deal with overlap (conflicts) between layers.
+            </note>
+        </para>
+    </section>
+
+    <section id='bb-bitbake-providers'>
+        <title>Providers</title>
+
+        <para>
+            Assuming BitBake has been instructed to execute a target
+            and that all the recipe files have been parsed, BitBake
+            starts to figure out how to build the target.
+            BitBake looks through the <filename>PROVIDES</filename> list
+            for each of the recipes.
+            A <filename>PROVIDES</filename> list is the list of names by which
+            the recipe can be known.
+            Each recipe's <filename>PROVIDES</filename> list is created
+            implicitly through the recipe's
+            <link linkend='var-PN'><filename>PN</filename></link> variable
+            and explicitly through the recipe's
+            <link linkend='var-PROVIDES'><filename>PROVIDES</filename></link>
+            variable, which is optional.
+        </para>
+
+        <para>
+            When a recipe uses <filename>PROVIDES</filename>, that recipe's
+            functionality can be found under an alternative name or names other
+            than the implicit <filename>PN</filename> name.
+            As an example, suppose a recipe named <filename>keyboard_1.0.bb</filename>
+            contained the following:
+            <literallayout class='monospaced'>
+     PROVIDES += "fullkeyboard"
+            </literallayout>
+            The <filename>PROVIDES</filename> list for this recipe becomes
+            "keyboard", which is implicit, and "fullkeyboard", which is explicit.
+            Consequently, the functionality found in
+            <filename>keyboard_1.0.bb</filename> can be found under two
+            different names.
+        </para>
+    </section>
+
+    <section id='bb-bitbake-preferences'>
+        <title>Preferences</title>
+
+        <para>
+            The <filename>PROVIDES</filename> list is only part of the solution
+            for figuring out a target's recipes.
+            Because targets might have multiple providers, BitBake needs
+            to prioritize providers by determining provider preferences.
+        </para>
+
+        <para>
+            A common example in which a target has multiple providers
+            is "virtual/kernel", which is on the
+            <filename>PROVIDES</filename> list for each kernel recipe.
+            Each machine often selects the best kernel provider by using a
+            line similar to the following in the machine configuration file:
+            <literallayout class='monospaced'>
+     PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
+            </literallayout>
+            The default
+            <link linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
+            is the provider with the same name as the target.
+            Bitbake iterates through each target it needs to build and
+            resolves them and their dependencies using this process.
+        </para>
+
+        <para>
+            Understanding how providers are chosen is made complicated by the fact
+            that multiple versions might exist for a given provider.
+            BitBake defaults to the highest version of a provider.
+            Version comparisons are made using the same method as Debian.
+            You can use the
+            <link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
+            variable to specify a particular version.
+            You can influence the order by using the
+            <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
+            variable.
+        </para>
+
+        <para>
+            By default, files have a preference of "0".
+            Setting <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
+            recipe unlikely to be used unless it is explicitly referenced.
+            Setting <filename>DEFAULT_PREFERENCE</filename> to "1" makes it
+            likely the recipe is used.
+            <filename>PREFERRED_VERSION</filename> overrides any
+            <filename>DEFAULT_PREFERENCE</filename> setting.
+            <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer
+            and more experimental recipe versions until they have undergone
+            sufficient testing to be considered stable.
+        </para>
+
+        <para>
+            When there are multiple “versions” of a given recipe,
+            BitBake defaults to selecting the most recent
+            version, unless otherwise specified.
+            If the recipe in question has a
+            <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
+            set lower than the other recipes (default is 0), then
+            it will not be selected.
+            This allows the person or persons maintaining
+            the repository of recipe files to specify
+            their preference for the default selected version.
+            Additionally, the user can specify their preferred version.
+        </para>
+
+        <para>
+            If the first recipe is named <filename>a_1.1.bb</filename>, then the
+            <link linkend='var-PN'><filename>PN</filename></link> variable
+            will be set to “a”, and the
+            <link linkend='var-PV'><filename>PV</filename></link>
+            variable will be set to 1.1.
+        </para>
+
+        <para>
+            Thus, if a recipe named <filename>a_1.2.bb</filename> exists, BitBake
+            will choose 1.2 by default.
+            However, if you define the following variable in a
+            <filename>.conf</filename> file that BitBake parses, you
+            can change that preference:
+            <literallayout class='monospaced'>
+     PREFERRED_VERSION_a = "1.1"
+            </literallayout>
+        </para>
+
+        <note>
+            <para>
+                It is common for a recipe to provide two versions -- a stable,
+                numbered (and preferred) version, and a version that is
+                automatically checked out from a source code repository that
+                is considered more "bleeding edge" but can be selected only
+                explicitly.
+            </para>
+
+            <para>
+                For example, in the OpenEmbedded codebase, there is a standard,
+                versioned recipe file for BusyBox,
+                <filename>busybox_1.22.1.bb</filename>,
+                but there is also a Git-based version,
+                <filename>busybox_git.bb</filename>, which explicitly contains the line
+                <literallayout class='monospaced'>
+    DEFAULT_PREFERENCE = "-1"
+                </literallayout>
+                to ensure that the numbered, stable version is always preferred
+                unless the developer selects otherwise.
+            </para>
+        </note>
+    </section>
+
+    <section id='bb-bitbake-dependencies'>
+        <title>Dependencies</title>
+
+        <para>
+            Each target BitBake builds consists of multiple tasks such as
+            <filename>fetch</filename>, <filename>unpack</filename>,
+            <filename>patch</filename>, <filename>configure</filename>,
+            and <filename>compile</filename>.
+            For best performance on multi-core systems, BitBake considers each
+            task as an independent
+            entity with its own set of dependencies.
+        </para>
+
+        <para>
+            Dependencies are defined through several variables.
+            You can find information about variables BitBake uses in
+            the <link linkend='ref-variables-glos'>Variables Glossary</link>
+            near the end of this manual.
+            At a basic level, it is sufficient to know that BitBake uses the
+            <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link> and
+            <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link> variables when
+            calculating dependencies.
+        </para>
+
+        <para>
+            For more information on how BitBake handles dependencies, see the
+            "<link linkend='dependencies'>Dependencies</link>" section.
+        </para>
+    </section>
+
+    <section id='ref-bitbake-tasklist'>
+        <title>The Task List</title>
+
+        <para>
+            Based on the generated list of providers and the dependency information,
+            BitBake can now calculate exactly what tasks it needs to run and in what
+            order it needs to run them.
+            The
+            "<link linkend='executing-tasks'>Executing Tasks</link>" section has more
+            information on how BitBake chooses which task to execute next.
+        </para>
+
+        <para>
+            The build now starts with BitBake forking off threads up to the limit set in the
+            <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
+            variable.
+            BitBake continues to fork threads as long as there are tasks ready to run,
+            those tasks have all their dependencies met, and the thread threshold has not been
+            exceeded.
+        </para>
+
+        <para>
+            It is worth noting that you can greatly speed up the build time by properly setting
+            the <filename>BB_NUMBER_THREADS</filename> variable.
+        </para>
+
+        <para>
+            As each task completes, a timestamp is written to the directory specified by the
+            <link linkend='var-STAMP'><filename>STAMP</filename></link> variable.
+            On subsequent runs, BitBake looks in the build directory within
+            <filename>tmp/stamps</filename> and does not rerun
+            tasks that are already completed unless a timestamp is found to be invalid.
+            Currently, invalid timestamps are only considered on a per
+            recipe file basis.
+            So, for example, if the configure stamp has a timestamp greater than the
+            compile timestamp for a given target, then the compile task would rerun.
+            Running the compile task again, however, has no effect on other providers
+            that depend on that target.
+        </para>
+
+        <para>
+            The exact format of the stamps is partly configurable.
+            In modern versions of BitBake, a hash is appended to the
+            stamp so that if the configuration changes, the stamp becomes
+            invalid and the task is automatically rerun.
+            This hash, or signature used, is governed by the signature policy
+            that is configured (see the
+            "<link linkend='checksums'>Checksums (Signatures)</link>"
+            section for information).
+            It is also possible to append extra metadata to the stamp using
+            the <filename>[stamp-extra-info]</filename> task flag.
+            For example, OpenEmbedded uses this flag to make some tasks machine-specific.
+        </para>
+
+        <note>
+            Some tasks are marked as "nostamp" tasks.
+            No timestamp file is created when these tasks are run.
+            Consequently, "nostamp" tasks are always rerun.
+        </note>
+
+        <para>
+            For more information on tasks, see the
+            "<link linkend='tasks'>Tasks</link>" section.
+        </para>
+    </section>
+
+    <section id='executing-tasks'>
+        <title>Executing Tasks</title>
+
+        <para>
+            Tasks can be either a shell task or a Python task.
+            For shell tasks, BitBake writes a shell script to
+            <filename>${</filename><link linkend='var-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename>
+            and then executes the script.
+            The generated shell script contains all the exported variables,
+            and the shell functions with all variables expanded.
+            Output from the shell script goes to the file
+            <filename>${T}/log.do_taskname.pid</filename>.
+            Looking at the expanded shell functions in the run file and
+            the output in the log files is a useful debugging technique.
+        </para>
+
+        <para>
+            For Python tasks, BitBake executes the task internally and logs
+            information to the controlling terminal.
+            Future versions of BitBake will write the functions to files
+            similar to the way shell tasks are handled.
+            Logging will be handled in a way similar to shell tasks as well.
+        </para>
+
+        <para>
+            The order in which BitBake runs the tasks is controlled by its
+            task scheduler.
+            It is possible to configure the scheduler and define custom
+            implementations for specific use cases.
+            For more information, see these variables that control the
+            behavior:
+            <itemizedlist>
+                <listitem><para>
+                    <link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
+                    </para></listitem>
+                <listitem><para>
+                    <link linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
+                    </para></listitem>
+            </itemizedlist>
+            It is possible to have functions run before and after a task's main
+            function.
+            This is done using the <filename>[prefuncs]</filename>
+            and <filename>[postfuncs]</filename> flags of the task
+            that lists the functions to run.
+        </para>
+    </section>
+
+    <section id='checksums'>
+        <title>Checksums (Signatures)</title>
+
+        <para>
+            A checksum is a unique signature of a task's inputs.
+            The signature of a task can be used to determine if a task
+            needs to be run.
+            Because it is a change in a task's inputs that triggers running
+            the task, BitBake needs to detect all the inputs to a given task.
+            For shell tasks, this turns out to be fairly easy because
+            BitBake generates a "run" shell script for each task and
+            it is possible to create a checksum that gives you a good idea of when
+            the task's data changes.
+        </para>
+
+        <para>
+            To complicate the problem, some things should not be included in
+            the checksum.
+            First, there is the actual specific build path of a given task -
+            the working directory.
+            It does not matter if the working directory changes because it should not
+            affect the output for target packages.
+            The simplistic approach for excluding the working directory is to set
+            it to some fixed value and create the checksum for the "run" script.
+            BitBake goes one step better and uses the
+            <link linkend='var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
+            variable to define a list of variables that should never be included
+            when generating the signatures.
+        </para>
+
+        <para>
+            Another problem results from the "run" scripts containing functions that
+            might or might not get called.
+            The incremental build solution contains code that figures out dependencies
+            between shell functions.
+            This code is used to prune the "run" scripts down to the minimum set,
+            thereby alleviating this problem and making the "run" scripts much more
+            readable as a bonus.
+        </para>
+
+        <para>
+            So far we have solutions for shell scripts.
+            What about Python tasks?
+            The same approach applies even though these tasks are more difficult.
+            The process needs to figure out what variables a Python function accesses
+            and what functions it calls.
+            Again, the incremental build solution contains code that first figures out
+            the variable and function dependencies, and then creates a checksum for the data
+            used as the input to the task.
+        </para>
+
+        <para>
+            Like the working directory case, situations exist where dependencies
+            should be ignored.
+            For these cases, you can instruct the build process to ignore a dependency
+            by using a line like the following:
+            <literallayout class='monospaced'>
+     PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
+            </literallayout>
+            This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not
+            depend on the value of <filename>MACHINE</filename>, even if it does reference it.
+        </para>
+
+        <para>
+            Equally, there are cases where we need to add dependencies BitBake
+            is not able to find.
+            You can accomplish this by using a line like the following:
+            <literallayout class='monospaced'>
+      PACKAGE_ARCHS[vardeps] = "MACHINE"
+            </literallayout>
+            This example explicitly adds the <filename>MACHINE</filename> variable as a
+            dependency for <filename>PACKAGE_ARCHS</filename>.
+        </para>
+
+        <para>
+            Consider a case with in-line Python, for example, where BitBake is not
+            able to figure out dependencies.
+            When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
+            produces output when it discovers something for which it cannot figure out
+            dependencies.
+        </para>
+
+        <para>
+            Thus far, this section has limited discussion to the direct inputs into a task.
+            Information based on direct inputs is referred to as the "basehash" in the
+            code.
+            However, there is still the question of a task's indirect inputs - the
+            things that were already built and present in the build directory.
+            The checksum (or signature) for a particular task needs to add the hashes
+            of all the tasks on which the particular task depends.
+            Choosing which dependencies to add is a policy decision.
+            However, the effect is to generate a master checksum that combines the basehash
+            and the hashes of the task's dependencies.
+        </para>
+
+        <para>
+            At the code level, there are a variety of ways both the basehash and the
+            dependent task hashes can be influenced.
+            Within the BitBake configuration file, we can give BitBake some extra information
+            to help it construct the basehash.
+            The following statement effectively results in a list of global variable
+            dependency excludes - variables never included in any checksum.
+            This example uses variables from OpenEmbedded to help illustrate
+            the concept:
+            <literallayout class='monospaced'>
+     BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
+         SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
+         USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
+         PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
+         CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
+            </literallayout>
+            The previous example excludes the work directory, which is part of
+            <filename>TMPDIR</filename>.
+        </para>
+
+        <para>
+            The rules for deciding which hashes of dependent tasks to include through
+            dependency chains are more complex and are generally accomplished with a
+            Python function.
+            The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
+            of this and also illustrates how you can insert your own policy into the system
+            if so desired.
+            This file defines the two basic signature generators OpenEmbedded Core
+            uses:  "OEBasic" and "OEBasicHash".
+            By default, there is a dummy "noop" signature handler enabled in BitBake.
+            This means that behavior is unchanged from previous versions.
+            <filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default
+            through this setting in the <filename>bitbake.conf</filename> file:
+            <literallayout class='monospaced'>
+     BB_SIGNATURE_HANDLER ?= "OEBasicHash"
+            </literallayout>
+            The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
+            "OEBasic" version but adds the task hash to the stamp files.
+            This results in any metadata change that changes the task hash, automatically
+            causing the task to be run again.
+            This removes the need to bump
+            <link linkend='var-PR'><filename>PR</filename></link>
+            values, and changes to metadata automatically ripple across the build.
+        </para>
+
+        <para>
+            It is also worth noting that the end result of these signature generators is to
+            make some dependency and hash information available to the build.
+            This information includes:
+            <itemizedlist>
+                <listitem><para><filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
+                    The base hashes for each task in the recipe.
+                    </para></listitem>
+                <listitem><para><filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
+                    The base hashes for each dependent task.
+                    </para></listitem>
+                <listitem><para><filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
+                    The task dependencies for each task.
+                    </para></listitem>
+                <listitem><para><filename>BB_TASKHASH</filename>:
+                    The hash of the currently running task.
+                    </para></listitem>
+            </itemizedlist>
+        </para>
+
+        <para>
+            It is worth noting that BitBake's "-S" option lets you
+            debug Bitbake's processing of signatures.
+            The options passed to -S allow different debugging modes
+            to be used, either using BitBake's own debug functions
+            or possibly those defined in the metadata/signature handler
+            itself.
+            The simplest parameter to pass is "none", which causes a
+            set of signature information to be written out into
+            <filename>STAMPS_DIR</filename>
+            corresponding to the targets specified.
+            The other currently available parameter is "printdiff",
+            which causes BitBake to try to establish the closest
+            signature match it can (e.g. in the sstate cache) and then
+            run <filename>bitbake-diffsigs</filename> over the matches
+            to determine the stamps and delta where these two
+            stamp trees diverge.
+            <note>
+                It is likely that future versions of BitBake will
+                provide other signature handlers triggered through
+                additional "-S" parameters.
+            </note>
+        </para>
+
+        <para>
+            You can find more information on checksum metadata in the
+            "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
+            section.
+        </para>
+    </section>
+
+    <section id='setscene'>
+        <title>Setscene</title>
+
+        <para>
+            The setscene process enables BitBake to handle "pre-built" artifacts.
+            The ability to handle and reuse these artifacts allows BitBake
+            the luxury of not having to build something from scratch every time.
+            Instead, BitBake can use, when possible, existing build artifacts.
+        </para>
+
+        <para>
+            BitBake needs to have reliable data indicating whether or not an
+            artifact is compatible.
+            Signatures, described in the previous section, provide an ideal
+            way of representing whether an artifact is compatible.
+            If a signature is the same, an object can be reused.
+        </para>
+
+        <para>
+            If an object can be reused, the problem then becomes how to
+            replace a given task or set of tasks with the pre-built artifact.
+            BitBake solves the problem with the "setscene" process.
+        </para>
+
+        <para>
+            When BitBake is asked to build a given target, before building anything,
+            it first asks whether cached information is available for any of the
+            targets it's building, or any of the intermediate targets.
+            If cached information is available, BitBake uses this information instead of
+            running the main tasks.
+        </para>
+
+        <para>
+            BitBake first calls the function defined by the
+            <link linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
+            variable with a list of tasks and corresponding
+            hashes it wants to build.
+            This function is designed to be fast and returns a list
+            of the tasks for which it believes in can obtain artifacts.
+        </para>
+
+        <para>
+            Next, for each of the tasks that were returned as possibilities,
+            BitBake executes a setscene version of the task that the possible
+            artifact covers.
+            Setscene versions of a task have the string "_setscene" appended to the
+            task name.
+            So, for example, the task with the name <filename>xxx</filename> has
+            a setscene task named <filename>xxx_setscene</filename>.
+            The setscene version of the task executes and provides the necessary
+            artifacts returning either success or failure.
+        </para>
+
+        <para>
+            As previously mentioned, an artifact can cover more than one task.
+            For example, it is pointless to obtain a compiler if you
+            already have the compiled binary.
+            To handle this, BitBake calls the
+            <link linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
+            function for each successful setscene task to know whether or not it needs
+            to obtain the dependencies of that task.
+        </para>
+
+        <para>
+            Finally, after all the setscene tasks have executed, BitBake calls the
+            function listed in
+            <link linkend='var-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>
+            with the list of tasks BitBake thinks has been "covered".
+            The metadata can then ensure that this list is correct and can
+            inform BitBake that it wants specific tasks to be run regardless
+            of the setscene result.
+        </para>
+
+        <para>
+            You can find more information on setscene metadata in the
+            "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
+            section.
+        </para>
+    </section>
+</chapter>

bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst

@@ -1,621 +0,0 @@
-.. SPDX-License-Identifier: CC-BY-2.5
-
-=====================
-File Download Support
-=====================
-
-|
-
-BitBake's fetch module is a standalone piece of library code that deals
-with the intricacies of downloading source code and files from remote
-systems. Fetching source code is one of the cornerstones of building
-software. As such, this module forms an important part of BitBake.
-
-The current fetch module is called "fetch2" and refers to the fact that
-it is the second major version of the API. The original version is
-obsolete and has been removed from the codebase. Thus, in all cases,
-"fetch" refers to "fetch2" in this manual.
-
-The Download (Fetch)
-====================
-
-BitBake takes several steps when fetching source code or files. The
-fetcher codebase deals with two distinct processes in order: obtaining
-the files from somewhere (cached or otherwise) and then unpacking those
-files into a specific location and perhaps in a specific way. Getting
-and unpacking the files is often optionally followed by patching.
-Patching, however, is not covered by this module.
-
-The code to execute the first part of this process, a fetch, looks
-something like the following: ::
-
-   src_uri = (d.getVar('SRC_URI') or "").split()
-   fetcher = bb.fetch2.Fetch(src_uri, d)
-   fetcher.download()
-
-This code sets up an instance of the fetch class. The instance uses a
-space-separated list of URLs from the :term:`SRC_URI`
-variable and then calls the ``download`` method to download the files.
-
-The instantiation of the fetch class is usually followed by: ::
-
-   rootdir = l.getVar('WORKDIR')
-   fetcher.unpack(rootdir)
-
-This code unpacks the downloaded files to the specified by ``WORKDIR``.
-
-.. note::
-
-   For convenience, the naming in these examples matches the variables
-   used by OpenEmbedded. If you want to see the above code in action,
-   examine the OpenEmbedded class file ``base.bbclass``
-   .
-
-The ``SRC_URI`` and ``WORKDIR`` variables are not hardcoded into the
-fetcher, since those fetcher methods can be (and are) called with
-different variable names. In OpenEmbedded for example, the shared state
-(sstate) code uses the fetch module to fetch the sstate files.
-
-When the ``download()`` method is called, BitBake tries to resolve the
-URLs by looking for source files in a specific search order:
-
--  *Pre-mirror Sites:* BitBake first uses pre-mirrors to try and find
-   source files. These locations are defined using the
-   :term:`PREMIRRORS` variable.
-
--  *Source URI:* If pre-mirrors fail, BitBake uses the original URL (e.g
-   from ``SRC_URI``).
-
--  *Mirror Sites:* If fetch failures occur, BitBake next uses mirror
-   locations as defined by the :term:`MIRRORS` variable.
-
-For each URL passed to the fetcher, the fetcher calls the submodule that
-handles that particular URL type. This behavior can be the source of
-some confusion when you are providing URLs for the ``SRC_URI`` variable.
-Consider the following two URLs: ::
-
-   http://git.yoctoproject.org/git/poky;protocol=git
-   git://git.yoctoproject.org/git/poky;protocol=http
-
-In the former case, the URL is passed to the ``wget`` fetcher, which does not
-understand "git". Therefore, the latter case is the correct form since the Git
-fetcher does know how to use HTTP as a transport.
-
-Here are some examples that show commonly used mirror definitions: ::
-
-   PREMIRRORS ?= "\
-      bzr://.*/.\*  http://somemirror.org/sources/ \\n \
-      cvs://.*/.\*  http://somemirror.org/sources/ \\n \
-      git://.*/.\*  http://somemirror.org/sources/ \\n \
-      hg://.*/.\*   http://somemirror.org/sources/ \\n \
-      osc://.*/.\*  http://somemirror.org/sources/ \\n \
-      p4://.*/.\*   http://somemirror.org/sources/ \\n \
-     svn://.*/.\*   http://somemirror.org/sources/ \\n"
-
-   MIRRORS =+ "\
-      ftp://.*/.\*   http://somemirror.org/sources/ \\n \
-      http://.*/.\*  http://somemirror.org/sources/ \\n \
-      https://.*/.\* http://somemirror.org/sources/ \\n"
-
-It is useful to note that BitBake
-supports cross-URLs. It is possible to mirror a Git repository on an
-HTTP server as a tarball. This is what the ``git://`` mapping in the
-previous example does.
-
-Since network accesses are slow, BitBake maintains a cache of files
-downloaded from the network. Any source files that are not local (i.e.
-downloaded from the Internet) are placed into the download directory,
-which is specified by the :term:`DL_DIR` variable.
-
-File integrity is of key importance for reproducing builds. For
-non-local archive downloads, the fetcher code can verify SHA-256 and MD5
-checksums to ensure the archives have been downloaded correctly. You can
-specify these checksums by using the ``SRC_URI`` variable with the
-appropriate varflags as follows: ::
-
-   SRC_URI[md5sum] = "value"
-   SRC_URI[sha256sum] = "value"
-
-You can also specify the checksums as
-parameters on the ``SRC_URI`` as shown below: ::
-
-  SRC_URI = "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d"
-
-If multiple URIs exist, you can specify the checksums either directly as
-in the previous example, or you can name the URLs. The following syntax
-shows how you name the URIs: ::
-
-   SRC_URI = "http://example.com/foobar.tar.bz2;name=foo"
-   SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d
-
-After a file has been downloaded and
-has had its checksum checked, a ".done" stamp is placed in ``DL_DIR``.
-BitBake uses this stamp during subsequent builds to avoid downloading or
-comparing a checksum for the file again.
-
-.. note::
-
-   It is assumed that local storage is safe from data corruption. If
-   this were not the case, there would be bigger issues to worry about.
-
-If :term:`BB_STRICT_CHECKSUM` is set, any
-download without a checksum triggers an error message. The
-:term:`BB_NO_NETWORK` variable can be used to
-make any attempted network access a fatal error, which is useful for
-checking that mirrors are complete as well as other things.
-
-.. _bb-the-unpack:
-
-The Unpack
-==========
-
-The unpack process usually immediately follows the download. For all
-URLs except Git URLs, BitBake uses the common ``unpack`` method.
-
-A number of parameters exist that you can specify within the URL to
-govern the behavior of the unpack stage:
-
--  *unpack:* Controls whether the URL components are unpacked. If set to
-   "1", which is the default, the components are unpacked. If set to
-   "0", the unpack stage leaves the file alone. This parameter is useful
-   when you want an archive to be copied in and not be unpacked.
-
--  *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether
-   to use DOS line ending conversion on text files.
-
--  *basepath:* Instructs the unpack stage to strip the specified
-   directories from the source path when unpacking.
-
--  *subdir:* Unpacks the specific URL to the specified subdirectory
-   within the root directory.
-
-The unpack call automatically decompresses and extracts files with ".Z",
-".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and
-".bz2" extensions as well as various combinations of tarball extensions.
-
-As mentioned, the Git fetcher has its own unpack method that is
-optimized to work with Git trees. Basically, this method works by
-cloning the tree into the final directory. The process is completed
-using references so that there is only one central copy of the Git
-metadata needed.
-
-.. _bb-fetchers:
-
-Fetchers
-========
-
-As mentioned earlier, the URL prefix determines which fetcher submodule
-BitBake uses. Each submodule can support different URL parameters, which
-are described in the following sections.
-
-.. _local-file-fetcher:
-
-Local file fetcher (``file://``)
---------------------------------
-
-This submodule handles URLs that begin with ``file://``. The filename
-you specify within the URL can be either an absolute or relative path to
-a file. If the filename is relative, the contents of the
-:term:`FILESPATH` variable is used in the same way
-``PATH`` is used to find executables. If the file cannot be found, it is
-assumed that it is available in :term:`DL_DIR` by the
-time the ``download()`` method is called.
-
-If you specify a directory, the entire directory is unpacked.
-
-Here are a couple of example URLs, the first relative and the second
-absolute: ::
-
-   SRC_URI = "file://relativefile.patch"
-   SRC_URI = "file:///Users/ich/very_important_software"
-
-.. _http-ftp-fetcher:
-
-HTTP/FTP wget fetcher (``http://``, ``ftp://``, ``https://``)
--------------------------------------------------------------
-
-This fetcher obtains files from web and FTP servers. Internally, the
-fetcher uses the wget utility.
-
-The executable and parameters used are specified by the
-``FETCHCMD_wget`` variable, which defaults to sensible values. The
-fetcher supports a parameter "downloadfilename" that allows the name of
-the downloaded file to be specified. Specifying the name of the
-downloaded file is useful for avoiding collisions in
-:term:`DL_DIR` when dealing with multiple files that
-have the same name.
-
-Some example URLs are as follows: ::
-
-   SRC_URI = "http://oe.handhelds.org/not_there.aac"
-   SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac"
-   SRC_URI = "ftp://you@oe.handhelds.org/home/you/secret.plan"
-
-.. note::
-
-   Because URL parameters are delimited by semi-colons, this can
-   introduce ambiguity when parsing URLs that also contain semi-colons,
-   for example:
-   ::
-
-           SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"
-
-
-   Such URLs should should be modified by replacing semi-colons with '&'
-   characters:
-   ::
-
-           SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47"
-
-
-   In most cases this should work. Treating semi-colons and '&' in
-   queries identically is recommended by the World Wide Web Consortium
-   (W3C). Note that due to the nature of the URL, you may have to
-   specify the name of the downloaded file as well:
-   ::
-
-           SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2"
-
-
-.. _cvs-fetcher:
-
-CVS fetcher (``(cvs://``)
--------------------------
-
-This submodule handles checking out files from the CVS version control
-system. You can configure it using a number of different variables:
-
--  :term:`FETCHCMD_cvs <FETCHCMD>`: The name of the executable to use when running
-   the ``cvs`` command. This name is usually "cvs".
-
--  :term:`SRCDATE`: The date to use when fetching the CVS source code. A
-   special value of "now" causes the checkout to be updated on every
-   build.
-
--  :term:`CVSDIR`: Specifies where a temporary
-   checkout is saved. The location is often ``DL_DIR/cvs``.
-
--  CVS_PROXY_HOST: The name to use as a "proxy=" parameter to the
-   ``cvs`` command.
-
--  CVS_PROXY_PORT: The port number to use as a "proxyport="
-   parameter to the ``cvs`` command.
-
-As well as the standard username and password URL syntax, you can also
-configure the fetcher with various URL parameters:
-
-The supported parameters are as follows:
-
--  *"method":* The protocol over which to communicate with the CVS
-   server. By default, this protocol is "pserver". If "method" is set to
-   "ext", BitBake examines the "rsh" parameter and sets ``CVS_RSH``. You
-   can use "dir" for local directories.
-
--  *"module":* Specifies the module to check out. You must supply this
-   parameter.
-
--  *"tag":* Describes which CVS TAG should be used for the checkout. By
-   default, the TAG is empty.
-
--  *"date":* Specifies a date. If no "date" is specified, the
-   :term:`SRCDATE` of the configuration is used to
-   checkout a specific date. The special value of "now" causes the
-   checkout to be updated on every build.
-
--  *"localdir":* Used to rename the module. Effectively, you are
-   renaming the output directory to which the module is unpacked. You
-   are forcing the module into a special directory relative to
-   :term:`CVSDIR`.
-
--  *"rsh":* Used in conjunction with the "method" parameter.
-
--  *"scmdata":* Causes the CVS metadata to be maintained in the tarball
-   the fetcher creates when set to "keep". The tarball is expanded into
-   the work directory. By default, the CVS metadata is removed.
-
--  *"fullpath":* Controls whether the resulting checkout is at the
-   module level, which is the default, or is at deeper paths.
-
--  *"norecurse":* Causes the fetcher to only checkout the specified
-   directory with no recurse into any subdirectories.
-
--  *"port":* The port to which the CVS server connects.
-
-Some example URLs are as follows: ::
-
-   SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
-   SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
-
-.. _svn-fetcher:
-
-Subversion (SVN) Fetcher (``svn://``)
--------------------------------------
-
-This fetcher submodule fetches code from the Subversion source control
-system. The executable used is specified by ``FETCHCMD_svn``, which
-defaults to "svn". The fetcher's temporary working directory is set by
-:term:`SVNDIR`, which is usually ``DL_DIR/svn``.
-
-The supported parameters are as follows:
-
--  *"module":* The name of the svn module to checkout. You must provide
-   this parameter. You can think of this parameter as the top-level
-   directory of the repository data you want.
-
--  *"path_spec":* A specific directory in which to checkout the
-   specified svn module.
-
--  *"protocol":* The protocol to use, which defaults to "svn". If
-   "protocol" is set to "svn+ssh", the "ssh" parameter is also used.
-
--  *"rev":* The revision of the source code to checkout.
-
--  *"scmdata":* Causes the ".svn" directories to be available during
-   compile-time when set to "keep". By default, these directories are
-   removed.
-
--  *"ssh":* An optional parameter used when "protocol" is set to
-   "svn+ssh". You can use this parameter to specify the ssh program used
-   by svn.
-
--  *"transportuser":* When required, sets the username for the
-   transport. By default, this parameter is empty. The transport
-   username is different than the username used in the main URL, which
-   is passed to the subversion command.
-
-Following are three examples using svn: ::
-
-   SRC_URI = "svn://myrepos/proj1;module=vip;protocol=http;rev=667"
-   SRC_URI = "svn://myrepos/proj1;module=opie;protocol=svn+ssh"
-   SRC_URI = "svn://myrepos/proj1;module=trunk;protocol=http;path_spec=${MY_DIR}/proj1"
-
-.. _git-fetcher:
-
-Git Fetcher (``git://``)
-------------------------
-
-This fetcher submodule fetches code from the Git source control system.
-The fetcher works by creating a bare clone of the remote into
-:term:`GITDIR`, which is usually ``DL_DIR/git2``. This
-bare clone is then cloned into the work directory during the unpack
-stage when a specific tree is checked out. This is done using alternates
-and by reference to minimize the amount of duplicate data on the disk
-and make the unpack process fast. The executable used can be set with
-``FETCHCMD_git``.
-
-This fetcher supports the following parameters:
-
--  *"protocol":* The protocol used to fetch the files. The default is
-   "git" when a hostname is set. If a hostname is not set, the Git
-   protocol is "file". You can also use "http", "https", "ssh" and
-   "rsync".
-
--  *"nocheckout":* Tells the fetcher to not checkout source code when
-   unpacking when set to "1". Set this option for the URL where there is
-   a custom routine to checkout code. The default is "0".
-
--  *"rebaseable":* Indicates that the upstream Git repository can be
-   rebased. You should set this parameter to "1" if revisions can become
-   detached from branches. In this case, the source mirror tarball is
-   done per revision, which has a loss of efficiency. Rebasing the
-   upstream Git repository could cause the current revision to disappear
-   from the upstream repository. This option reminds the fetcher to
-   preserve the local cache carefully for future use. The default value
-   for this parameter is "0".
-
--  *"nobranch":* Tells the fetcher to not check the SHA validation for
-   the branch when set to "1". The default is "0". Set this option for
-   the recipe that refers to the commit that is valid for any namespace
-   (branch, tag, ...) instead of the branch.
-
--  *"bareclone":* Tells the fetcher to clone a bare clone into the
-   destination directory without checking out a working tree. Only the
-   raw Git metadata is provided. This parameter implies the "nocheckout"
-   parameter as well.
-
--  *"branch":* The branch(es) of the Git tree to clone. If unset, this
-   is assumed to be "master". The number of branch parameters much match
-   the number of name parameters.
-
--  *"rev":* The revision to use for the checkout. The default is
-   "master".
-
--  *"tag":* Specifies a tag to use for the checkout. To correctly
-   resolve tags, BitBake must access the network. For that reason, tags
-   are often not used. As far as Git is concerned, the "tag" parameter
-   behaves effectively the same as the "rev" parameter.
-
--  *"subpath":* Limits the checkout to a specific subpath of the tree.
-   By default, the whole tree is checked out.
-
--  *"destsuffix":* The name of the path in which to place the checkout.
-   By default, the path is ``git/``.
-
--  *"usehead":* Enables local ``git://`` URLs to use the current branch
-   HEAD as the revision for use with ``AUTOREV``. The "usehead"
-   parameter implies no branch and only works when the transfer protocol
-   is ``file://``.
-
-Here are some example URLs: ::
-
-   SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
-   SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http"
-
-.. _gitsm-fetcher:
-
-Git Submodule Fetcher (``gitsm://``)
-------------------------------------
-
-This fetcher submodule inherits from the :ref:`Git
-fetcher<bitbake-user-manual/bitbake-user-manual-fetching:git fetcher
-(\`\`git://\`\`)>` and extends that fetcher's behavior by fetching a
-repository's submodules. :term:`SRC_URI` is passed to the Git fetcher as
-described in the :ref:`bitbake-user-manual/bitbake-user-manual-fetching:git
-fetcher (\`\`git://\`\`)` section.
-
-.. note::
-
-   You must clean a recipe when switching between '``git://``' and
-   '``gitsm://``' URLs.
-
-   The Git Submodules fetcher is not a complete fetcher implementation.
-   The fetcher has known issues where it does not use the normal source
-   mirroring infrastructure properly. Further, the submodule sources it
-   fetches are not visible to the licensing and source archiving
-   infrastructures.
-
-.. _clearcase-fetcher:
-
-ClearCase Fetcher (``ccrc://``)
--------------------------------
-
-This fetcher submodule fetches code from a
-`ClearCase <http://en.wikipedia.org/wiki/Rational_ClearCase>`__
-repository.
-
-To use this fetcher, make sure your recipe has proper
-:term:`SRC_URI`, :term:`SRCREV`, and
-:term:`PV` settings. Here is an example: ::
-
-   SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
-   SRCREV = "EXAMPLE_CLEARCASE_TAG"
-   PV = "${@d.getVar("SRCREV", False).replace("/", "+")}"
-
-The fetcher uses the ``rcleartool`` or
-``cleartool`` remote client, depending on which one is available.
-
-Following are options for the ``SRC_URI`` statement:
-
--  *vob*: The name, which must include the prepending "/" character,
-   of the ClearCase VOB. This option is required.
-
--  *module*: The module, which must include the prepending "/"
-   character, in the selected VOB.
-
-   .. note::
-
-      The module and vob options are combined to create the load rule in the
-      view config spec. As an example, consider the vob and module values from
-      the SRC_URI statement at the start of this section. Combining those values
-      results in the following: ::
-
-         load /example_vob/example_module
-
--  *proto*: The protocol, which can be either ``http`` or ``https``.
-
-By default, the fetcher creates a configuration specification. If you
-want this specification written to an area other than the default, use
-the ``CCASE_CUSTOM_CONFIG_SPEC`` variable in your recipe to define where
-the specification is written.
-
-.. note::
-
-   the SRCREV loses its functionality if you specify this variable. However,
-   SRCREV is still used to label the archive after a fetch even though it does
-   not define what is fetched.
-
-Here are a couple of other behaviors worth mentioning:
-
--  When using ``cleartool``, the login of ``cleartool`` is handled by
-   the system. The login require no special steps.
-
--  In order to use ``rcleartool`` with authenticated users, an
-   "rcleartool login" is necessary before using the fetcher.
-
-.. _perforce-fetcher:
-
-Perforce Fetcher (``p4://``)
-----------------------------
-
-This fetcher submodule fetches code from the
-`Perforce <https://www.perforce.com/>`__ source control system. The
-executable used is specified by ``FETCHCMD_p4``, which defaults to "p4".
-The fetcher's temporary working directory is set by
-:term:`P4DIR`, which defaults to "DL_DIR/p4".
-The fetcher does not make use of a perforce client, instead it
-relies on ``p4 files`` to retrieve a list of
-files and ``p4 print`` to transfer the content
-of those files locally.
-
-To use this fetcher, make sure your recipe has proper
-:term:`SRC_URI`, :term:`SRCREV`, and
-:term:`PV` values. The p4 executable is able to use the
-config file defined by your system's ``P4CONFIG`` environment variable
-in order to define the Perforce server URL and port, username, and
-password if you do not wish to keep those values in a recipe itself. If
-you choose not to use ``P4CONFIG``, or to explicitly set variables that
-``P4CONFIG`` can contain, you can specify the ``P4PORT`` value, which is
-the server's URL and port number, and you can specify a username and
-password directly in your recipe within ``SRC_URI``.
-
-Here is an example that relies on ``P4CONFIG`` to specify the server URL
-and port, username, and password, and fetches the Head Revision: ::
-
-   SRC_URI = "p4://example-depot/main/source/..."
-   SRCREV = "${AUTOREV}"
-   PV = "p4-${SRCPV}"
-   S = "${WORKDIR}/p4"
-
-Here is an example that specifies the server URL and port, username, and
-password, and fetches a Revision based on a Label: ::
-
-   P4PORT = "tcp:p4server.example.net:1666"
-   SRC_URI = "p4://user:passwd@example-depot/main/source/..."
-   SRCREV = "release-1.0"
-   PV = "p4-${SRCPV}"
-   S = "${WORKDIR}/p4"
-
-.. note::
-
-   You should always set S to "${WORKDIR}/p4" in your recipe.
-
-.. _repo-fetcher:
-
-Repo Fetcher (``repo://``)
---------------------------
-
-This fetcher submodule fetches code from ``google-repo`` source control
-system. The fetcher works by initiating and syncing sources of the
-repository into :term:`REPODIR`, which is usually
-``${DL_DIR}/repo``.
-
-This fetcher supports the following parameters:
-
--  *"protocol":* Protocol to fetch the repository manifest (default:
-   git).
-
--  *"branch":* Branch or tag of repository to get (default: master).
-
--  *"manifest":* Name of the manifest file (default: ``default.xml``).
-
-Here are some example URLs: ::
-
-   SRC_URI = "repo://REPOROOT;protocol=git;branch=some_branch;manifest=my_manifest.xml"
-   SRC_URI = "repo://REPOROOT;protocol=file;branch=some_branch;manifest=my_manifest.xml"
-
-Other Fetchers
---------------
-
-Fetch submodules also exist for the following:
-
--  Bazaar (``bzr://``)
-
--  Mercurial (``hg://``)
-
--  npm (``npm://``)
-
--  OSC (``osc://``)
-
--  Secure FTP (``sftp://``)
-
--  Secure Shell (``ssh://``)
-
--  Trees using Git Annex (``gitannex://``)
-
-No documentation currently exists for these lesser used fetcher
-submodules. However, you might find the code helpful and readable.
-
-Auto Revisions
-==============
-
-We need to document ``AUTOREV`` and ``SRCREV_FORMAT`` here.

bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml

@@ -0,0 +1,823 @@
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+
+<chapter>
+<title>File Download Support</title>
+
+    <para>
+        BitBake's fetch module is a standalone piece of library code
+        that deals with the intricacies of downloading source code
+        and files from remote systems.
+        Fetching source code is one of the cornerstones of building software.
+        As such, this module forms an important part of BitBake.
+    </para>
+
+    <para>
+        The current fetch module is called "fetch2" and refers to the
+        fact that it is the second major version of the API.
+        The original version is obsolete and has been removed from the codebase.
+        Thus, in all cases, "fetch" refers to "fetch2" in this
+        manual.
+    </para>
+
+    <section id='the-download-fetch'>
+        <title>The Download (Fetch)</title>
+
+        <para>
+            BitBake takes several steps when fetching source code or files.
+            The fetcher codebase deals with two distinct processes in order:
+            obtaining the files from somewhere (cached or otherwise)
+            and then unpacking those files into a specific location and
+            perhaps in a specific way.
+            Getting and unpacking the files is often optionally followed
+            by patching.
+            Patching, however, is not covered by this module.
+        </para>
+
+        <para>
+            The code to execute the first part of this process, a fetch,
+            looks something like the following:
+            <literallayout class='monospaced'>
+     src_uri = (d.getVar('SRC_URI') or "").split()
+     fetcher = bb.fetch2.Fetch(src_uri, d)
+     fetcher.download()
+            </literallayout>
+            This code sets up an instance of the fetch class.
+            The instance uses a space-separated list of URLs from the
+            <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
+            variable and then calls the <filename>download</filename>
+            method to download the files.
+        </para>
+
+        <para>
+            The instantiation of the fetch class is usually followed by:
+            <literallayout class='monospaced'>
+     rootdir = l.getVar('WORKDIR')
+     fetcher.unpack(rootdir)
+            </literallayout>
+            This code unpacks the downloaded files to the
+            specified by <filename>WORKDIR</filename>.
+            <note>
+                For convenience, the naming in these examples matches
+                the variables used by OpenEmbedded.
+                If you want to see the above code in action, examine
+                the OpenEmbedded class file <filename>base.bbclass</filename>.
+            </note>
+            The <filename>SRC_URI</filename> and <filename>WORKDIR</filename>
+            variables are not hardcoded into the fetcher, since those fetcher
+            methods can be (and are) called with different variable names.
+            In OpenEmbedded for example, the shared state (sstate) code uses
+            the fetch module to fetch the sstate files.
+        </para>
+
+        <para>
+            When the <filename>download()</filename> method is called,
+            BitBake tries to resolve the URLs by looking for source files
+            in a specific search order:
+            <itemizedlist>
+                <listitem><para><emphasis>Pre-mirror Sites:</emphasis>
+                    BitBake first uses pre-mirrors to try and find source files.
+                    These locations are defined using the
+                    <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
+                    variable.
+                    </para></listitem>
+                <listitem><para><emphasis>Source URI:</emphasis>
+                    If pre-mirrors fail, BitBake uses the original URL (e.g from
+                    <filename>SRC_URI</filename>).
+                    </para></listitem>
+                <listitem><para><emphasis>Mirror Sites:</emphasis>
+                    If fetch failures occur, BitBake next uses mirror locations as
+                    defined by the
+                    <link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
+                    variable.
+                    </para></listitem>
+            </itemizedlist>
+        </para>
+
+        <para>
+            For each URL passed to the fetcher, the fetcher
+            calls the submodule that handles that particular URL type.
+            This behavior can be the source of some confusion when you
+            are providing URLs for the <filename>SRC_URI</filename>
+            variable.
+            Consider the following two URLs:
+            <literallayout class='monospaced'>
+     http://git.yoctoproject.org/git/poky;protocol=git
+     git://git.yoctoproject.org/git/poky;protocol=http
+            </literallayout>
+            In the former case, the URL is passed to the
+            <filename>wget</filename> fetcher, which does not
+            understand "git".
+            Therefore, the latter case is the correct form since the
+            Git fetcher does know how to use HTTP as a transport.
+        </para>
+
+        <para>
+            Here are some examples that show commonly used mirror
+            definitions:
+            <literallayout class='monospaced'>
+     PREMIRRORS ?= "\
+         bzr://.*/.*   http://somemirror.org/sources/ \n \
+         cvs://.*/.*   http://somemirror.org/sources/ \n \
+         git://.*/.*   http://somemirror.org/sources/ \n \
+         hg://.*/.*    http://somemirror.org/sources/ \n \
+         osc://.*/.*   http://somemirror.org/sources/ \n \
+         p4://.*/.*    http://somemirror.org/sources/ \n \
+         svn://.*/.*   http://somemirror.org/sources/ \n"
+
+     MIRRORS =+ "\
+         ftp://.*/.*      http://somemirror.org/sources/ \n \
+         http://.*/.*     http://somemirror.org/sources/ \n \
+         https://.*/.*    http://somemirror.org/sources/ \n"
+            </literallayout>
+            It is useful to note that BitBake supports
+            cross-URLs.
+            It is possible to mirror a Git repository on an HTTP
+            server as a tarball.
+            This is what the <filename>git://</filename> mapping in
+            the previous example does.
+        </para>
+
+        <para>
+            Since network accesses are slow, Bitbake maintains a
+            cache of files downloaded from the network.
+            Any source files that are not local (i.e.
+            downloaded from the Internet) are placed into the download
+            directory, which is specified by the
+            <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
+            variable.
+        </para>
+
+        <para>
+            File integrity is of key importance for reproducing builds.
+            For non-local archive downloads, the fetcher code can verify
+            SHA-256 and MD5 checksums to ensure the archives have been
+            downloaded correctly.
+            You can specify these checksums by using the
+            <filename>SRC_URI</filename> variable with the appropriate
+            varflags as follows:
+            <literallayout class='monospaced'>
+     SRC_URI[md5sum] = "<replaceable>value</replaceable>"
+     SRC_URI[sha256sum] = "<replaceable>value</replaceable>"
+            </literallayout>
+            You can also specify the checksums as parameters on the
+            <filename>SRC_URI</filename> as shown below:
+            <literallayout class='monospaced'>
+     SRC_URI = "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d"
+            </literallayout>
+            If multiple URIs exist, you can specify the checksums either
+            directly as in the previous example, or you can name the URLs.
+            The following syntax shows how you name the URIs:
+            <literallayout class='monospaced'>
+     SRC_URI = "http://example.com/foobar.tar.bz2;name=foo"
+     SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d
+            </literallayout>
+            After a file has been downloaded and has had its checksum checked,
+            a ".done" stamp is placed in <filename>DL_DIR</filename>.
+            BitBake uses this stamp during subsequent builds to avoid
+            downloading or comparing a checksum for the file again.
+            <note>
+                It is assumed that local storage is safe from data corruption.
+                If this were not the case, there would be bigger issues to worry about.
+            </note>
+        </para>
+
+        <para>
+            If
+            <link linkend='var-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
+            is set, any download without a checksum triggers an
+            error message.
+            The
+            <link linkend='var-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
+            variable can be used to make any attempted network access a fatal
+            error, which is useful for checking that mirrors are complete
+            as well as other things.
+        </para>
+    </section>
+
+    <section id='bb-the-unpack'>
+        <title>The Unpack</title>
+
+        <para>
+            The unpack process usually immediately follows the download.
+            For all URLs except Git URLs, BitBake uses the common
+            <filename>unpack</filename> method.
+        </para>
+
+        <para>
+            A number of parameters exist that you can specify within the
+            URL to govern the behavior of the unpack stage:
+            <itemizedlist>
+                <listitem><para><emphasis>unpack:</emphasis>
+                    Controls whether the URL components are unpacked.
+                    If set to "1", which is the default, the components
+                    are unpacked.
+                    If set to "0", the unpack stage leaves the file alone.
+                    This parameter is useful when you want an archive to be
+                    copied in and not be unpacked.
+                    </para></listitem>
+                <listitem><para><emphasis>dos:</emphasis>
+                    Applies to <filename>.zip</filename> and
+                    <filename>.jar</filename> files and specifies whether to
+                    use DOS line ending conversion on text files.
+                    </para></listitem>
+                <listitem><para><emphasis>basepath:</emphasis>
+                    Instructs the unpack stage to strip the specified
+                    directories from the source path when unpacking.
+                    </para></listitem>
+                <listitem><para><emphasis>subdir:</emphasis>
+                    Unpacks the specific URL to the specified subdirectory
+                    within the root directory.
+                    </para></listitem>
+            </itemizedlist>
+            The unpack call automatically decompresses and extracts files
+            with ".Z", ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm".
+            ".srpm", ".deb" and ".bz2" extensions as well as various combinations
+            of tarball extensions.
+        </para>
+
+        <para>
+            As mentioned, the Git fetcher has its own unpack method that
+            is optimized to work with Git trees.
+            Basically, this method works by cloning the tree into the final
+            directory.
+            The process is completed using references so that there is
+            only one central copy of the Git metadata needed.
+        </para>
+    </section>
+
+    <section id='bb-fetchers'>
+        <title>Fetchers</title>
+
+        <para>
+            As mentioned earlier, the URL prefix determines which
+            fetcher submodule BitBake uses.
+            Each submodule can support different URL parameters,
+            which are described in the following sections.
+        </para>
+
+        <section id='local-file-fetcher'>
+            <title>Local file fetcher (<filename>file://</filename>)</title>
+
+            <para>
+                This submodule handles URLs that begin with
+                <filename>file://</filename>.
+                The filename you specify within the URL can be
+                either an absolute or relative path to a file.
+                If the filename is relative, the contents of the
+                <link linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
+                variable is used in the same way
+                <filename>PATH</filename> is used to find executables.
+                If the file cannot be found, it is assumed that it is available in
+                <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
+                by the time the <filename>download()</filename> method is called.
+            </para>
+
+            <para>
+                If you specify a directory, the entire directory is
+                unpacked.
+            </para>
+
+            <para>
+                Here are a couple of example URLs, the first relative and
+                the second absolute:
+                <literallayout class='monospaced'>
+     SRC_URI = "file://relativefile.patch"
+     SRC_URI = "file:///Users/ich/very_important_software"
+                </literallayout>
+            </para>
+        </section>
+
+        <section id='http-ftp-fetcher'>
+            <title>HTTP/FTP wget fetcher (<filename>http://</filename>, <filename>ftp://</filename>, <filename>https://</filename>)</title>
+
+            <para>
+                This fetcher obtains files from web and FTP servers.
+                Internally, the fetcher uses the wget utility.
+            </para>
+
+            <para>
+                The executable and parameters used are specified by the
+                <filename>FETCHCMD_wget</filename> variable, which defaults
+                to sensible values.
+                The fetcher supports a parameter "downloadfilename" that
+                allows the name of the downloaded file to be specified.
+                Specifying the name of the downloaded file is useful
+                for avoiding collisions in
+                <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
+                when dealing with multiple files that have the same name.
+            </para>
+
+            <para>
+                Some example URLs are as follows:
+                <literallayout class='monospaced'>
+     SRC_URI = "http://oe.handhelds.org/not_there.aac"
+     SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac"
+     SRC_URI = "ftp://you@oe.handhelds.org/home/you/secret.plan"
+                </literallayout>
+            </para>
+            <note>
+               Because URL parameters are delimited by semi-colons, this can
+               introduce ambiguity when parsing URLs that also contain semi-colons,
+               for example:
+                <literallayout class='monospaced'>
+     SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"
+                </literallayout>
+               Such URLs should should be modified by replacing semi-colons with '&amp;' characters:
+               <literallayout class='monospaced'>
+     SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&amp;a=snapshot&amp;h=a5dd47"
+                </literallayout>
+                In most cases this should work. Treating semi-colons and '&amp;' in queries
+                identically is recommended by the World Wide Web Consortium (W3C).
+                Note that due to the nature of the URL, you may have to specify the name
+                of the downloaded file as well:
+              <literallayout class='monospaced'>
+     SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&amp;a=snapshot&amp;h=a5dd47;downloadfilename=myfile.bz2"
+                </literallayout>
+            </note>
+        </section>
+
+        <section id='cvs-fetcher'>
+            <title>CVS fetcher (<filename>(cvs://</filename>)</title>
+
+            <para>
+                This submodule handles checking out files from the
+                CVS version control system.
+                You can configure it using a number of different variables:
+                <itemizedlist>
+                    <listitem><para><emphasis><filename>FETCHCMD_cvs</filename>:</emphasis>
+                        The name of the executable to use when running
+                        the <filename>cvs</filename> command.
+                        This name is usually "cvs".
+                        </para></listitem>
+                    <listitem><para><emphasis><filename>SRCDATE</filename>:</emphasis>
+                        The date to use when fetching the CVS source code.
+                        A special value of "now" causes the checkout to
+                        be updated on every build.
+                        </para></listitem>
+                    <listitem><para><emphasis><link linkend='var-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
+                        Specifies where a temporary checkout is saved.
+                        The location is often <filename>DL_DIR/cvs</filename>.
+                        </para></listitem>
+                    <listitem><para><emphasis><filename>CVS_PROXY_HOST</filename>:</emphasis>
+                        The name to use as a "proxy=" parameter to the
+                        <filename>cvs</filename> command.
+                        </para></listitem>
+                    <listitem><para><emphasis><filename>CVS_PROXY_PORT</filename>:</emphasis>
+                        The port number to use as a "proxyport=" parameter to
+                        the <filename>cvs</filename> command.
+                        </para></listitem>
+                </itemizedlist>
+                As well as the standard username and password URL syntax,
+                you can also configure the fetcher with various URL parameters:
+            </para>
+
+            <para>
+                The supported parameters are as follows:
+                <itemizedlist>
+                    <listitem><para><emphasis>"method":</emphasis>
+                        The protocol over which to communicate with the CVS
+                        server.
+                        By default, this protocol is "pserver".
+                        If "method" is set to "ext", BitBake examines the
+                        "rsh" parameter and sets <filename>CVS_RSH</filename>.
+                        You can use "dir" for local directories.
+                        </para></listitem>
+                    <listitem><para><emphasis>"module":</emphasis>
+                        Specifies the module to check out.
+                        You must supply this parameter.
+                        </para></listitem>
+                    <listitem><para><emphasis>"tag":</emphasis>
+                        Describes which CVS TAG should be used for
+                        the checkout.
+                        By default, the TAG is empty.
+                        </para></listitem>
+                    <listitem><para><emphasis>"date":</emphasis>
+                        Specifies a date.
+                        If no "date" is specified, the
+                        <link linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
+                        of the configuration is used to checkout a specific date.
+                        The special value of "now" causes the checkout to be
+                        updated on every build.
+                        </para></listitem>
+                    <listitem><para><emphasis>"localdir":</emphasis>
+                        Used to rename the module.
+                        Effectively, you are renaming the output directory
+                        to which the module is unpacked.
+                        You are forcing the module into a special
+                        directory relative to
+                        <link linkend='var-CVSDIR'><filename>CVSDIR</filename></link>.
+                        </para></listitem>
+                    <listitem><para><emphasis>"rsh"</emphasis>
+                        Used in conjunction with the "method" parameter.
+                        </para></listitem>
+                    <listitem><para><emphasis>"scmdata":</emphasis>
+                        Causes the CVS metadata to be maintained in the tarball
+                        the fetcher creates when set to "keep".
+                        The tarball is expanded into the work directory.
+                        By default, the CVS metadata is removed.
+                        </para></listitem>
+                    <listitem><para><emphasis>"fullpath":</emphasis>
+                        Controls whether the resulting checkout is at the
+                        module level, which is the default, or is at deeper
+                        paths.
+                        </para></listitem>
+                    <listitem><para><emphasis>"norecurse":</emphasis>
+                        Causes the fetcher to only checkout the specified
+                        directory with no recurse into any subdirectories.
+                        </para></listitem>
+                    <listitem><para><emphasis>"port":</emphasis>
+                        The port to which the CVS server connects.
+                        </para></listitem>
+                </itemizedlist>
+                Some example URLs are as follows:
+                <literallayout class='monospaced'>
+     SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
+     SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
+                </literallayout>
+            </para>
+        </section>
+
+        <section id='svn-fetcher'>
+            <title>Subversion (SVN) Fetcher (<filename>svn://</filename>)</title>
+
+            <para>
+                This fetcher submodule fetches code from the
+                Subversion source control system.
+                The executable used is specified by
+                <filename>FETCHCMD_svn</filename>, which defaults
+                to "svn".
+                The fetcher's temporary working directory is set by
+                <link linkend='var-SVNDIR'><filename>SVNDIR</filename></link>,
+                which is usually <filename>DL_DIR/svn</filename>.
+            </para>
+
+            <para>
+                The supported parameters are as follows:
+                <itemizedlist>
+                    <listitem><para><emphasis>"module":</emphasis>
+                        The name of the svn module to checkout.
+                        You must provide this parameter.
+                        You can think of this parameter as the top-level
+                        directory of the repository data you want.
+                        </para></listitem>
+                    <listitem><para><emphasis>"path_spec":</emphasis>
+                        A specific directory in which to checkout the
+                        specified svn module.
+                        </para></listitem>
+                    <listitem><para><emphasis>"protocol":</emphasis>
+                        The protocol to use, which defaults to "svn".
+                        If "protocol" is set to "svn+ssh", the "ssh"
+                        parameter is also used.
+                        </para></listitem>
+                    <listitem><para><emphasis>"rev":</emphasis>
+                        The revision of the source code to checkout.
+                        </para></listitem>
+                    <listitem><para><emphasis>"scmdata":</emphasis>
+                        Causes the “.svn” directories to be available during
+                        compile-time when set to "keep".
+                        By default, these directories are removed.
+                        </para></listitem>
+                    <listitem><para><emphasis>"ssh":</emphasis>
+                        An optional parameter used when "protocol" is set
+                        to "svn+ssh".
+                        You can use this parameter to specify the ssh
+                        program used by svn.
+                        </para></listitem>
+                    <listitem><para><emphasis>"transportuser":</emphasis>
+                        When required, sets the username for the transport.
+                        By default, this parameter is empty.
+                        The transport username is different than the username
+                        used in the main URL, which is passed to the subversion
+                        command.
+                        </para></listitem>
+                </itemizedlist>
+                Following are three examples using svn:
+                <literallayout class='monospaced'>
+     SRC_URI = "svn://myrepos/proj1;module=vip;protocol=http;rev=667"
+     SRC_URI = "svn://myrepos/proj1;module=opie;protocol=svn+ssh"
+     SRC_URI = "svn://myrepos/proj1;module=trunk;protocol=http;path_spec=${MY_DIR}/proj1"
+                </literallayout>
+            </para>
+        </section>
+
+        <section id='git-fetcher'>
+            <title>Git Fetcher (<filename>git://</filename>)</title>
+
+            <para>
+                This fetcher submodule fetches code from the Git
+                source control system.
+                The fetcher works by creating a bare clone of the
+                remote into
+                <link linkend='var-GITDIR'><filename>GITDIR</filename></link>,
+                which is usually <filename>DL_DIR/git2</filename>.
+                This bare clone is then cloned into the work directory during the
+                unpack stage when a specific tree is checked out.
+                This is done using alternates and by reference to
+                minimize the amount of duplicate data on the disk and
+                make the unpack process fast.
+                The executable used can be set with
+                <filename>FETCHCMD_git</filename>.
+            </para>
+
+            <para>
+                This fetcher supports the following parameters:
+                <itemizedlist>
+                    <listitem><para><emphasis>"protocol":</emphasis>
+                        The protocol used to fetch the files.
+                        The default is "git" when a hostname is set.
+                        If a hostname is not set, the Git protocol is "file".
+                        You can also use "http", "https", "ssh" and "rsync".
+                        </para></listitem>
+                    <listitem><para><emphasis>"nocheckout":</emphasis>
+                        Tells the fetcher to not checkout source code when
+                        unpacking when set to "1".
+                        Set this option for the URL where there is a custom
+                        routine to checkout code.
+                        The default is "0".
+                        </para></listitem>
+                    <listitem><para><emphasis>"rebaseable":</emphasis>
+                        Indicates that the upstream Git repository can be rebased.
+                        You should set this parameter to "1" if
+                        revisions can become detached from branches.
+                        In this case, the source mirror tarball is done per
+                        revision, which has a loss of efficiency.
+                        Rebasing the upstream Git repository could cause the
+                        current revision to disappear from the upstream repository.
+                        This option reminds the fetcher to preserve the local cache
+                        carefully for future use.
+                        The default value for this parameter is "0".
+                        </para></listitem>
+                    <listitem><para><emphasis>"nobranch":</emphasis>
+                        Tells the fetcher to not check the SHA validation
+                        for the branch when set to "1".
+                        The default is "0".
+                        Set this option for the recipe that refers to
+                        the commit that is valid for a tag instead of
+                        the branch.
+                        </para></listitem>
+                    <listitem><para><emphasis>"bareclone":</emphasis>
+                        Tells the fetcher to clone a bare clone into the
+                        destination directory without checking out a working tree.
+                        Only the raw Git metadata is provided.
+                        This parameter implies the "nocheckout" parameter as well.
+                        </para></listitem>
+                    <listitem><para><emphasis>"branch":</emphasis>
+                        The branch(es) of the Git tree to clone.
+                        If unset, this is assumed to be "master".
+                        The number of branch parameters much match the number of
+                        name parameters.
+                        </para></listitem>
+                    <listitem><para><emphasis>"rev":</emphasis>
+                        The revision to use for the checkout.
+                        The default is "master".
+                        </para></listitem>
+                    <listitem><para><emphasis>"tag":</emphasis>
+                        Specifies a tag to use for the checkout.
+                        To correctly resolve tags, BitBake must access the
+                        network.
+                        For that reason, tags are often not used.
+                        As far as Git is concerned, the "tag" parameter behaves
+                        effectively the same as the "rev" parameter.
+                        </para></listitem>
+                    <listitem><para><emphasis>"subpath":</emphasis>
+                        Limits the checkout to a specific subpath of the tree.
+                        By default, the whole tree is checked out.
+                        </para></listitem>
+                    <listitem><para><emphasis>"destsuffix":</emphasis>
+                        The name of the path in which to place the checkout.
+                        By default, the path is <filename>git/</filename>.
+                        </para></listitem>
+                </itemizedlist>
+                Here are some example URLs:
+                <literallayout class='monospaced'>
+     SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
+     SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http"
+                </literallayout>
+            </para>
+        </section>
+
+        <section id='gitsm-fetcher'>
+            <title>Git Submodule Fetcher (<filename>gitsm://</filename>)</title>
+
+            <para>
+                This fetcher submodule inherits from the
+                <link linkend='git-fetcher'>Git fetcher</link> and extends
+                that fetcher's behavior by fetching a repository's submodules.
+                <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
+                is passed to the Git fetcher as described in the
+                "<link linkend='git-fetcher'>Git Fetcher (<filename>git://</filename>)</link>"
+                section.
+                <note>
+                    <title>Notes and Warnings</title>
+                    <para>
+                        You must clean a recipe when switching between
+                        '<filename>git://</filename>' and
+                        '<filename>gitsm://</filename>' URLs.
+                    </para>
+
+                    <para>
+                        The Git Submodules fetcher is not a complete fetcher
+                        implementation.
+                        The fetcher has known issues where it does not use the
+                        normal source mirroring infrastructure properly. Further,
+                        the submodule sources it fetches are not visible to the
+                        licensing and source archiving infrastructures.
+                    </para>
+                </note>
+            </para>
+        </section>
+
+        <section id='clearcase-fetcher'>
+            <title>ClearCase Fetcher (<filename>ccrc://</filename>)</title>
+
+            <para>
+                This fetcher submodule fetches code from a
+                <ulink url='http://en.wikipedia.org/wiki/Rational_ClearCase'>ClearCase</ulink>
+                repository.
+            </para>
+
+            <para>
+                To use this fetcher, make sure your recipe has proper
+                <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
+                <link linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
+                <link linkend='var-PV'><filename>PV</filename></link> settings.
+                Here is an example:
+                <literallayout class='monospaced'>
+     SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
+     SRCREV = "EXAMPLE_CLEARCASE_TAG"
+     PV = "${@d.getVar("SRCREV", False).replace("/", "+")}"
+                </literallayout>
+                The fetcher uses the <filename>rcleartool</filename> or
+                <filename>cleartool</filename> remote client, depending on
+                which one is available.
+            </para>
+
+            <para>
+                Following are options for the <filename>SRC_URI</filename>
+                statement:
+                <itemizedlist>
+                    <listitem><para><emphasis><filename>vob</filename></emphasis>:
+                        The name, which must include the
+                        prepending "/" character, of the ClearCase VOB.
+                        This option is required.
+                        </para></listitem>
+                    <listitem><para><emphasis><filename>module</filename></emphasis>:
+                        The module, which must include the
+                        prepending "/" character, in the selected VOB.
+                        <note>
+                            The <filename>module</filename> and <filename>vob</filename>
+                            options are combined to create the <filename>load</filename> rule in
+                            the view config spec.
+                            As an example, consider the <filename>vob</filename> and
+                            <filename>module</filename> values from the
+                            <filename>SRC_URI</filename> statement at the start of this section.
+                            Combining those values results in the following:
+                            <literallayout class='monospaced'>
+     load /example_vob/example_module
+                            </literallayout>
+                        </note>
+                        </para></listitem>
+                    <listitem><para><emphasis><filename>proto</filename></emphasis>:
+                        The protocol, which can be either <filename>http</filename> or
+                        <filename>https</filename>.
+                        </para></listitem>
+                </itemizedlist>
+            </para>
+
+            <para>
+                By default, the fetcher creates a configuration specification.
+                If you want this specification written to an area other than the default,
+                use the <filename>CCASE_CUSTOM_CONFIG_SPEC</filename> variable
+                in your recipe to define where the specification is written.
+                <note>
+                    the <filename>SRCREV</filename> loses its functionality if you
+                    specify this variable.
+                    However, <filename>SRCREV</filename> is still used to label the
+                    archive after a fetch even though it does not define what is
+                    fetched.
+                </note>
+            </para>
+
+            <para>
+                Here are a couple of other behaviors worth mentioning:
+                <itemizedlist>
+                    <listitem><para>
+                        When using <filename>cleartool</filename>, the login of
+                        <filename>cleartool</filename> is handled by the system.
+                        The login require no special steps.
+                        </para></listitem>
+                    <listitem><para>
+                        In order to use <filename>rcleartool</filename> with authenticated
+                        users, an "rcleartool login" is necessary before using the fetcher.
+                        </para></listitem>
+                </itemizedlist>
+            </para>
+        </section>
+
+        <section id='perforce-fetcher'>
+            <title>Perforce Fetcher (<filename>p4://</filename>)</title>
+
+            <para>
+                This fetcher submodule fetches code from the
+                <ulink url='https://www.perforce.com/'>Perforce</ulink>
+                source control system.
+                The executable used is specified by
+                <filename>FETCHCMD_p4</filename>, which defaults
+                to "p4".
+                The fetcher's temporary working directory is set by
+                <link linkend='var-P4DIR'><filename>P4DIR</filename></link>,
+                which defaults to "DL_DIR/p4".
+            </para>
+
+            <para>
+                To use this fetcher, make sure your recipe has proper
+                <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
+                <link linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
+                <link linkend='var-PV'><filename>PV</filename></link> values.
+                The p4 executable is able to use the config file defined by your
+                system's <filename>P4CONFIG</filename> environment variable in
+                order to define the Perforce server URL and port, username, and
+                password if you do not wish to keep those values in a recipe
+                itself.
+                If you choose not to use <filename>P4CONFIG</filename>,
+                or to explicitly set variables that <filename>P4CONFIG</filename>
+                can contain, you can specify the <filename>P4PORT</filename> value,
+                which is the server's URL and port number, and you can
+                specify a username and password directly in your recipe within
+                <filename>SRC_URI</filename>.
+            </para>
+
+            <para>
+                Here is an example that relies on <filename>P4CONFIG</filename>
+                to specify the server URL and port, username, and password, and
+                fetches the Head Revision:
+                <literallayout class='monospaced'>
+    SRC_URI = "p4://example-depot/main/source/..."
+    SRCREV = "${AUTOREV}"
+    PV = "p4-${SRCPV}"
+    S = "${WORKDIR}/p4"
+                </literallayout>
+            </para>
+
+            <para>
+                Here is an example that specifies the server URL and port,
+                username, and password, and fetches a Revision based on a Label:
+                <literallayout class='monospaced'>
+    P4PORT = "tcp:p4server.example.net:1666"
+    SRC_URI = "p4://user:passwd@example-depot/main/source/..."
+    SRCREV = "release-1.0"
+    PV = "p4-${SRCPV}"
+    S = "${WORKDIR}/p4"
+                </literallayout>
+                <note>
+                    You should always set <filename>S</filename>
+                    to <filename>"${WORKDIR}/p4"</filename> in your recipe.
+                </note>
+            </para>
+        </section>
+
+        <section id='other-fetchers'>
+            <title>Other Fetchers</title>
+
+            <para>
+                Fetch submodules also exist for the following:
+                <itemizedlist>
+                    <listitem><para>
+                        Bazaar (<filename>bzr://</filename>)
+                        </para></listitem>
+                    <listitem><para>
+                        Trees using Git Annex (<filename>gitannex://</filename>)
+                        </para></listitem>
+                    <listitem><para>
+                        Secure FTP (<filename>sftp://</filename>)
+                        </para></listitem>
+                    <listitem><para>
+                        Secure Shell (<filename>ssh://</filename>)
+                        </para></listitem>
+                    <listitem><para>
+                        Repo (<filename>repo://</filename>)
+                        </para></listitem>
+                    <listitem><para>
+                        OSC (<filename>osc://</filename>)
+                        </para></listitem>
+                    <listitem><para>
+                        Mercurial (<filename>hg://</filename>)
+                        </para></listitem>
+                </itemizedlist>
+                No documentation currently exists for these lesser used
+                fetcher submodules.
+                However, you might find the code helpful and readable.
+            </para>
+        </section>
+    </section>
+
+    <section id='auto-revisions'>
+        <title>Auto Revisions</title>
+
+        <para>
+            We need to document <filename>AUTOREV</filename> and
+            <filename>SRCREV_FORMAT</filename> here.
+        </para>
+    </section>
+</chapter>

bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.rst

@@ -1,415 +0,0 @@
-.. SPDX-License-Identifier: CC-BY-2.5
-
-===================
-Hello World Example
-===================
-
-BitBake Hello World
-===================
-
-The simplest example commonly used to demonstrate any new programming
-language or tool is the "`Hello
-World <http://en.wikipedia.org/wiki/Hello_world_program>`__" example.
-This appendix demonstrates, in tutorial form, Hello World within the
-context of BitBake. The tutorial describes how to create a new project
-and the applicable metadata files necessary to allow BitBake to build
-it.
-
-Obtaining BitBake
-=================
-
-See the :ref:`bitbake-user-manual/bitbake-user-manual-hello:obtaining bitbake` section for
-information on how to obtain BitBake. Once you have the source code on
-your machine, the BitBake directory appears as follows: ::
-
-   $ ls -al
-   total 100
-   drwxrwxr-x. 9 wmat wmat  4096 Jan 31 13:44 .
-   drwxrwxr-x. 3 wmat wmat  4096 Feb  4 10:45 ..
-   -rw-rw-r--. 1 wmat wmat   365 Nov 26 04:55 AUTHORS
-   drwxrwxr-x. 2 wmat wmat  4096 Nov 26 04:55 bin
-   drwxrwxr-x. 4 wmat wmat  4096 Jan 31 13:44 build
-   -rw-rw-r--. 1 wmat wmat 16501 Nov 26 04:55 ChangeLog
-   drwxrwxr-x. 2 wmat wmat  4096 Nov 26 04:55 classes
-   drwxrwxr-x. 2 wmat wmat  4096 Nov 26 04:55 conf
-   drwxrwxr-x. 3 wmat wmat  4096 Nov 26 04:55 contrib
-   -rw-rw-r--. 1 wmat wmat 17987 Nov 26 04:55 COPYING
-   drwxrwxr-x. 3 wmat wmat  4096 Nov 26 04:55 doc
-   -rw-rw-r--. 1 wmat wmat    69 Nov 26 04:55 .gitignore
-   -rw-rw-r--. 1 wmat wmat   849 Nov 26 04:55 HEADER
-   drwxrwxr-x. 5 wmat wmat  4096 Jan 31 13:44 lib
-   -rw-rw-r--. 1 wmat wmat   195 Nov 26 04:55 MANIFEST.in
-   -rw-rw-r--. 1 wmat wmat  2887 Nov 26 04:55 TODO
-
-At this point, you should have BitBake cloned to a directory that
-matches the previous listing except for dates and user names.
-
-Setting Up the BitBake Environment
-==================================
-
-First, you need to be sure that you can run BitBake. Set your working
-directory to where your local BitBake files are and run the following
-command: ::
-
-  $ ./bin/bitbake --version
-  BitBake Build Tool Core version 1.23.0, bitbake version 1.23.0
-
-The console output tells you what version
-you are running.
-
-The recommended method to run BitBake is from a directory of your
-choice. To be able to run BitBake from any directory, you need to add
-the executable binary to your binary to your shell's environment
-``PATH`` variable. First, look at your current ``PATH`` variable by
-entering the following: ::
-
-  $ echo $PATH
-
-Next, add the directory location
-for the BitBake binary to the ``PATH``. Here is an example that adds the
-``/home/scott-lenovo/bitbake/bin`` directory to the front of the
-``PATH`` variable: ::
-
-  $ export PATH=/home/scott-lenovo/bitbake/bin:$PATH
-
-You should now be able to enter the ``bitbake`` command from the command
-line while working from any directory.
-
-The Hello World Example
-=======================
-
-The overall goal of this exercise is to build a complete "Hello World"
-example utilizing task and layer concepts. Because this is how modern
-projects such as OpenEmbedded and the Yocto Project utilize BitBake, the
-example provides an excellent starting point for understanding BitBake.
-
-To help you understand how to use BitBake to build targets, the example
-starts with nothing but the ``bitbake`` command, which causes BitBake to
-fail and report problems. The example progresses by adding pieces to the
-build to eventually conclude with a working, minimal "Hello World"
-example.
-
-While every attempt is made to explain what is happening during the
-example, the descriptions cannot cover everything. You can find further
-information throughout this manual. Also, you can actively participate
-in the :oe_lists:`/g/bitbake-devel`
-discussion mailing list about the BitBake build tool.
-
-.. note::
-
-   This example was inspired by and drew heavily from
-   `Mailing List post - The BitBake equivalent of "Hello, World!"
-   <http://www.mail-archive.com/yocto@yoctoproject.org/msg09379.html>`_.
-
-As stated earlier, the goal of this example is to eventually compile
-"Hello World". However, it is unknown what BitBake needs and what you
-have to provide in order to achieve that goal. Recall that BitBake
-utilizes three types of metadata files:
-:ref:`bitbake-user-manual/bitbake-user-manual-intro:configuration files`,
-:ref:`bitbake-user-manual/bitbake-user-manual-intro:classes`, and
-:ref:`bitbake-user-manual/bitbake-user-manual-intro:recipes`.
-But where do they go? How does BitBake find
-them? BitBake's error messaging helps you answer these types of
-questions and helps you better understand exactly what is going on.
-
-Following is the complete "Hello World" example.
-
-#.  **Create a Project Directory:** First, set up a directory for the
-    "Hello World" project. Here is how you can do so in your home
-    directory: ::
-
-      $ mkdir ~/hello
-      $ cd ~/hello
-
-    This is the directory that
-    BitBake will use to do all of its work. You can use this directory
-    to keep all the metafiles needed by BitBake. Having a project
-    directory is a good way to isolate your project.
-
-#.  **Run BitBake:** At this point, you have nothing but a project
-    directory. Run the ``bitbake`` command and see what it does: ::
-
-       $ bitbake
-       The BBPATH variable is not set and bitbake did not
-       find a conf/bblayers.conf file in the expected location.
-       Maybe you accidentally invoked bitbake from the wrong directory?
-       DEBUG: Removed the following variables from the environment:
-       GNOME_DESKTOP_SESSION_ID, XDG_CURRENT_DESKTOP,
-       GNOME_KEYRING_CONTROL, DISPLAY, SSH_AGENT_PID, LANG, no_proxy,
-       XDG_SESSION_PATH, XAUTHORITY, SESSION_MANAGER, SHLVL,
-       MANDATORY_PATH, COMPIZ_CONFIG_PROFILE, WINDOWID, EDITOR,
-       GPG_AGENT_INFO, SSH_AUTH_SOCK, GDMSESSION, GNOME_KEYRING_PID,
-       XDG_SEAT_PATH, XDG_CONFIG_DIRS, LESSOPEN, DBUS_SESSION_BUS_ADDRESS,
-       _, XDG_SESSION_COOKIE, DESKTOP_SESSION, LESSCLOSE, DEFAULTS_PATH,
-       UBUNTU_MENUPROXY, OLDPWD, XDG_DATA_DIRS, COLORTERM, LS_COLORS
-
-    The majority of this output is specific to environment variables that
-    are not directly relevant to BitBake. However, the very first
-    message regarding the ``BBPATH`` variable and the
-    ``conf/bblayers.conf`` file is relevant.
-
-    When you run BitBake, it begins looking for metadata files. The
-    :term:`BBPATH` variable is what tells BitBake where
-    to look for those files. ``BBPATH`` is not set and you need to set
-    it. Without ``BBPATH``, BitBake cannot find any configuration files
-    (``.conf``) or recipe files (``.bb``) at all. BitBake also cannot
-    find the ``bitbake.conf`` file.
-
-#.  **Setting BBPATH:** For this example, you can set ``BBPATH`` in
-    the same manner that you set ``PATH`` earlier in the appendix. You
-    should realize, though, that it is much more flexible to set the
-    ``BBPATH`` variable up in a configuration file for each project.
-
-    From your shell, enter the following commands to set and export the
-    ``BBPATH`` variable: ::
-
-      $ BBPATH="projectdirectory"
-      $ export BBPATH
-
-    Use your actual project directory in the command. BitBake uses that
-    directory to find the metadata it needs for your project.
-
-    .. note::
-
-       When specifying your project directory, do not use the tilde
-       ("~") character as BitBake does not expand that character as the
-       shell would.
-
-#.  **Run BitBake:** Now that you have ``BBPATH`` defined, run the
-    ``bitbake`` command again: ::
-
-       $ bitbake
-       ERROR: Traceback (most recent call last):
-         File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 163, in wrapped
-           return func(fn, *args)
-         File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 173, in parse_config_file
-           return bb.parse.handle(fn, data, include)
-         File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line 99, in handle
-           return h['handle'](fn, data, include)
-         File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 120, in handle
-           abs_fn = resolve_file(fn, data)
-         File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line 117, in resolve_file
-           raise IOError("file %s not found in %s" % (fn, bbpath))
-       IOError: file conf/bitbake.conf not found in /home/scott-lenovo/hello
-
-       ERROR: Unable to parse conf/bitbake.conf: file conf/bitbake.conf not found in /home/scott-lenovo/hello
-
-    This sample output shows that BitBake could not find the
-    ``conf/bitbake.conf`` file in the project directory. This file is
-    the first thing BitBake must find in order to build a target. And,
-    since the project directory for this example is empty, you need to
-    provide a ``conf/bitbake.conf`` file.
-
-#.  **Creating conf/bitbake.conf:** The ``conf/bitbake.conf`` includes
-    a number of configuration variables BitBake uses for metadata and
-    recipe files. For this example, you need to create the file in your
-    project directory and define some key BitBake variables. For more
-    information on the ``bitbake.conf`` file, see
-    http://git.openembedded.org/bitbake/tree/conf/bitbake.conf.
-
-    Use the following commands to create the ``conf`` directory in the
-    project directory: ::
-
-      $ mkdir conf
-
-    From within the ``conf`` directory,
-    use some editor to create the ``bitbake.conf`` so that it contains
-    the following: ::
-
-       PN  = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
-
-       TMPDIR  = "${TOPDIR}/tmp"
-       CACHE   = "${TMPDIR}/cache"
-       STAMP   = "${TMPDIR}/${PN}/stamps"
-       T       = "${TMPDIR}/${PN}/work"
-       B       = "${TMPDIR}/${PN}"
-
-    .. note::
-
-       Without a value for PN , the variables STAMP , T , and B , prevent more
-       than one recipe from working. You can fix this by either setting PN to
-       have a value similar to what OpenEmbedded and BitBake use in the default
-       bitbake.conf file (see previous example). Or, by manually updating each
-       recipe to set PN . You will also need to include PN as part of the STAMP
-       , T , and B variable definitions in the local.conf file.
-
-    The ``TMPDIR`` variable establishes a directory that BitBake uses
-    for build output and intermediate files other than the cached
-    information used by the
-    :ref:`bitbake-user-manual/bitbake-user-manual-execution:setscene`
-    process. Here, the ``TMPDIR`` directory is set to ``hello/tmp``.
-
-    .. tip::
-
-       You can always safely delete the tmp directory in order to rebuild a
-       BitBake target. The build process creates the directory for you when you
-       run BitBake.
-
-    For information about each of the other variables defined in this
-    example, check :term:`PN`, :term:`TOPDIR`, :term:`CACHE`, :term:`STAMP`,
-    :term:`T` or :term:`B` to take you to the definitions in the
-    glossary.
-
-#.  **Run BitBake:** After making sure that the ``conf/bitbake.conf`` file
-    exists, you can run the ``bitbake`` command again: ::
-
-       $ bitbake
-       ERROR: Traceback (most recent call last):
-         File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 163, in wrapped
-           return func(fn, *args)
-         File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 177, in _inherit
-           bb.parse.BBHandler.inherit(bbclass, "configuration INHERITs", 0, data)
-         File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 92, in inherit
-           include(fn, file, lineno, d, "inherit")
-         File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 100, in include
-           raise ParseError("Could not %(error_out)s file %(fn)s" % vars(), oldfn, lineno)
-       ParseError: ParseError in configuration INHERITs: Could not inherit file classes/base.bbclass
-
-       ERROR: Unable to parse base: ParseError in configuration INHERITs: Could not inherit file classes/base.bbclass
-
-    In the sample output,
-    BitBake could not find the ``classes/base.bbclass`` file. You need
-    to create that file next.
-
-#.  **Creating classes/base.bbclass:** BitBake uses class files to
-    provide common code and functionality. The minimally required class
-    for BitBake is the ``classes/base.bbclass`` file. The ``base`` class
-    is implicitly inherited by every recipe. BitBake looks for the class
-    in the ``classes`` directory of the project (i.e ``hello/classes``
-    in this example).
-
-    Create the ``classes`` directory as follows: ::
-
-      $ cd $HOME/hello
-      $ mkdir classes
-
-    Move to the ``classes`` directory and then create the
-    ``base.bbclass`` file by inserting this single line: addtask build
-    The minimal task that BitBake runs is the ``do_build`` task. This is
-    all the example needs in order to build the project. Of course, the
-    ``base.bbclass`` can have much more depending on which build
-    environments BitBake is supporting.
-
-#.  **Run BitBake:** After making sure that the ``classes/base.bbclass``
-    file exists, you can run the ``bitbake`` command again: ::
-
-       $ bitbake
-       Nothing to do. Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.
-
-    BitBake is finally reporting
-    no errors. However, you can see that it really does not have
-    anything to do. You need to create a recipe that gives BitBake
-    something to do.
-
-#.  **Creating a Layer:** While it is not really necessary for such a
-    small example, it is good practice to create a layer in which to
-    keep your code separate from the general metadata used by BitBake.
-    Thus, this example creates and uses a layer called "mylayer".
-
-    .. note::
-
-       You can find additional information on layers in the
-       ":ref:`bitbake-user-manual/bitbake-user-manual-intro:Layers`" section.
-
-    Minimally, you need a recipe file and a layer configuration file in
-    your layer. The configuration file needs to be in the ``conf``
-    directory inside the layer. Use these commands to set up the layer
-    and the ``conf`` directory: ::
-
-       $ cd $HOME
-       $ mkdir mylayer
-       $ cd mylayer
-       $ mkdir conf
-
-    Move to the ``conf`` directory and create a ``layer.conf`` file that has the
-    following: ::
-
-      BBPATH .= ":${LAYERDIR}"
-      BBFILES += "${LAYERDIR}/\*.bb"
-      BBFILE_COLLECTIONS += "mylayer"
-     `BBFILE_PATTERN_mylayer := "^${LAYERDIR_RE}/"
-
-    For information on these variables, click on :term:`BBFILES`,
-    :term:`LAYERDIR`, :term:`BBFILE_COLLECTIONS` or :term:`BBFILE_PATTERN_mylayer <BBFILE_PATTERN>`
-    to go to the definitions in the glossary.
-
-    You need to create the recipe file next. Inside your layer at the
-    top-level, use an editor and create a recipe file named
-    ``printhello.bb`` that has the following: ::
-
-       DESCRIPTION = "Prints Hello World"
-       PN = 'printhello'
-       PV = '1'
-
-       python do_build() {
-          bb.plain("********************");
-          bb.plain("*                  *");
-          bb.plain("*  Hello, World!   *");
-          bb.plain("*                  *");
-          bb.plain("********************");
-       }
-
-    The recipe file simply provides
-    a description of the recipe, the name, version, and the ``do_build``
-    task, which prints out "Hello World" to the console. For more
-    information on :term:`DESCRIPTION`, :term:`PN` or :term:`PV`
-    follow the links to the glossary.
-
-#. **Run BitBake With a Target:** Now that a BitBake target exists, run
-    the command and provide that target: ::
-
-      $ cd $HOME/hello
-      $ bitbake printhello
-      ERROR: no recipe files to build, check your BBPATH and BBFILES?
-
-      Summary: There was 1 ERROR message shown, returning a non-zero exit code.
-
-    We have created the layer with the recipe and
-    the layer configuration file but it still seems that BitBake cannot
-    find the recipe. BitBake needs a ``conf/bblayers.conf`` that lists
-    the layers for the project. Without this file, BitBake cannot find
-    the recipe.
-
-#. **Creating conf/bblayers.conf:** BitBake uses the
-    ``conf/bblayers.conf`` file to locate layers needed for the project.
-    This file must reside in the ``conf`` directory of the project (i.e.
-    ``hello/conf`` for this example).
-
-    Set your working directory to the ``hello/conf`` directory and then
-    create the ``bblayers.conf`` file so that it contains the following: ::
-
-       BBLAYERS ?= " \
-           /home/<you>/mylayer \
-       "
-
-    You need to provide your own information for ``you`` in the file.
-
-#. **Run BitBake With a Target:** Now that you have supplied the
-    ``bblayers.conf`` file, run the ``bitbake`` command and provide the
-    target: ::
-
-       $ bitbake printhello
-       Parsing recipes: 100% |##################################################################################|
-       Time: 00:00:00
-       Parsing of 1 .bb files complete (0 cached, 1 parsed). 1 targets, 0 skipped, 0 masked, 0 errors.
-       NOTE: Resolving any missing task queue dependencies
-       NOTE: Preparing RunQueue
-       NOTE: Executing RunQueue Tasks
-       ********************
-       *                  *
-       *  Hello, World!   *
-       *                  *
-       ********************
-       NOTE: Tasks Summary: Attempted 1 tasks of which 0 didn't need to be rerun and all succeeded.
-
-    .. note::
-
-       After the first execution, re-running bitbake printhello again will not
-       result in a BitBake run that prints the same console output. The reason
-       for this is that the first time the printhello.bb recipe's do_build task
-       executes successfully, BitBake writes a stamp file for the task. Thus,
-       the next time you attempt to run the task using that same bitbake
-       command, BitBake notices the stamp and therefore determines that the task
-       does not need to be re-run. If you delete the tmp directory or run
-       bitbake -c clean printhello and then re-run the build, the "Hello,
-       World!" message will be printed again.

bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml

@@ -0,0 +1,495 @@
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+
+<appendix id='hello-world-example'>
+    <title>Hello World Example</title>
+
+    <section id='bitbake-hello-world'>
+        <title>BitBake Hello World</title>
+
+        <para>
+            The simplest example commonly used to demonstrate any new
+            programming language or tool is the
+            "<ulink url="http://en.wikipedia.org/wiki/Hello_world_program">Hello World</ulink>"
+            example.
+            This appendix demonstrates, in tutorial form, Hello
+            World within the context of BitBake.
+            The tutorial describes how to create a new project
+            and the applicable metadata files necessary to allow
+            BitBake to build it.
+        </para>
+    </section>
+
+    <section id='example-obtaining-bitbake'>
+        <title>Obtaining BitBake</title>
+
+        <para>
+            See the
+            "<link linkend='obtaining-bitbake'>Obtaining BitBake</link>"
+            section for information on how to obtain BitBake.
+            Once you have the source code on your machine, the BitBake directory
+            appears as follows:
+            <literallayout class='monospaced'>
+     $ ls -al
+     total 100
+     drwxrwxr-x. 9 wmat wmat  4096 Jan 31 13:44 .
+     drwxrwxr-x. 3 wmat wmat  4096 Feb  4 10:45 ..
+     -rw-rw-r--. 1 wmat wmat   365 Nov 26 04:55 AUTHORS
+     drwxrwxr-x. 2 wmat wmat  4096 Nov 26 04:55 bin
+     drwxrwxr-x. 4 wmat wmat  4096 Jan 31 13:44 build
+     -rw-rw-r--. 1 wmat wmat 16501 Nov 26 04:55 ChangeLog
+     drwxrwxr-x. 2 wmat wmat  4096 Nov 26 04:55 classes
+     drwxrwxr-x. 2 wmat wmat  4096 Nov 26 04:55 conf
+     drwxrwxr-x. 3 wmat wmat  4096 Nov 26 04:55 contrib
+     -rw-rw-r--. 1 wmat wmat 17987 Nov 26 04:55 COPYING
+     drwxrwxr-x. 3 wmat wmat  4096 Nov 26 04:55 doc
+     -rw-rw-r--. 1 wmat wmat    69 Nov 26 04:55 .gitignore
+     -rw-rw-r--. 1 wmat wmat   849 Nov 26 04:55 HEADER
+     drwxrwxr-x. 5 wmat wmat  4096 Jan 31 13:44 lib
+     -rw-rw-r--. 1 wmat wmat   195 Nov 26 04:55 MANIFEST.in
+     -rw-rw-r--. 1 wmat wmat  2887 Nov 26 04:55 TODO
+            </literallayout>
+        </para>
+
+        <para>
+            At this point, you should have BitBake cloned to
+            a directory that matches the previous listing except for
+            dates and user names.
+        </para>
+    </section>
+
+    <section id='setting-up-the-bitbake-environment'>
+        <title>Setting Up the BitBake Environment</title>
+
+        <para>
+            First, you need to be sure that you can run BitBake.
+            Set your working directory to where your local BitBake
+            files are and run the following command:
+            <literallayout class='monospaced'>
+     $ ./bin/bitbake --version
+     BitBake Build Tool Core version 1.23.0, bitbake version 1.23.0
+            </literallayout>
+            The console output tells you what version you are running.
+        </para>
+
+        <para>
+            The recommended method to run BitBake is from a directory of your
+            choice.
+            To be able to run BitBake from any directory, you need to add the
+            executable binary to your binary to your shell's environment
+            <filename>PATH</filename> variable.
+            First, look at your current <filename>PATH</filename> variable
+            by entering the following:
+            <literallayout class='monospaced'>
+     $ echo $PATH
+            </literallayout>
+            Next, add the directory location for the BitBake binary to the
+            <filename>PATH</filename>.
+            Here is an example that adds the
+            <filename>/home/scott-lenovo/bitbake/bin</filename> directory
+            to the front of the <filename>PATH</filename> variable:
+            <literallayout class='monospaced'>
+     $ export PATH=/home/scott-lenovo/bitbake/bin:$PATH
+            </literallayout>
+            You should now be able to enter the <filename>bitbake</filename>
+            command from the command line while working from any directory.
+        </para>
+    </section>
+
+    <section id='the-hello-world-example'>
+        <title>The Hello World Example</title>
+
+        <para>
+            The overall goal of this exercise is to build a
+            complete "Hello World" example utilizing task and layer
+            concepts.
+            Because this is how modern projects such as OpenEmbedded and
+            the Yocto Project utilize BitBake, the example
+            provides an excellent starting point for understanding
+            BitBake.
+        </para>
+
+        <para>
+            To help you understand how to use BitBake to build targets,
+            the example starts with nothing but the <filename>bitbake</filename>
+            command, which causes BitBake to fail and report problems.
+            The example progresses by adding pieces to the build to
+            eventually conclude with a working, minimal "Hello World"
+            example.
+        </para>
+
+        <para>
+            While every attempt is made to explain what is happening during
+            the example, the descriptions cannot cover everything.
+            You can find further information throughout this manual.
+            Also, you can actively participate in the
+            <ulink url='http://lists.openembedded.org/mailman/listinfo/bitbake-devel'></ulink>
+            discussion mailing list about the BitBake build tool.
+        </para>
+
+        <note>
+            This example was inspired by and drew heavily from
+            <ulink url="http://www.mail-archive.com/yocto@yoctoproject.org/msg09379.html">Mailing List post - The BitBake equivalent of "Hello, World!"</ulink>.
+        </note>
+
+        <para>
+            As stated earlier, the goal of this example
+            is to eventually compile "Hello World".
+            However, it is unknown what BitBake needs and what you have
+            to provide in order to achieve that goal.
+            Recall that BitBake utilizes three types of metadata files:
+            <link linkend='configuration-files'>Configuration Files</link>,
+            <link linkend='classes'>Classes</link>, and
+            <link linkend='recipes'>Recipes</link>.
+            But where do they go?
+            How does BitBake find them?
+            BitBake's error messaging helps you answer these types of questions
+            and helps you better understand exactly what is going on.
+        </para>
+
+        <para>
+            Following is the complete "Hello World" example.
+        </para>
+
+        <orderedlist>
+            <listitem><para><emphasis>Create a Project Directory:</emphasis>
+                First, set up a directory for the "Hello World" project.
+                Here is how you can do so in your home directory:
+                <literallayout class='monospaced'>
+     $ mkdir ~/hello
+     $ cd ~/hello
+                </literallayout>
+                This is the directory that BitBake will use to do all of
+                its work.
+                You can use this directory to keep all the metafiles needed
+                by BitBake.
+                Having a project directory is a good way to isolate your
+                project.
+                </para></listitem>
+            <listitem><para><emphasis>Run Bitbake:</emphasis>
+                At this point, you have nothing but a project directory.
+                Run the <filename>bitbake</filename> command and see what
+                it does:
+                <literallayout class='monospaced'>
+     $ bitbake
+     The BBPATH variable is not set and bitbake did not
+     find a conf/bblayers.conf file in the expected location.
+     Maybe you accidentally invoked bitbake from the wrong directory?
+     DEBUG: Removed the following variables from the environment:
+     GNOME_DESKTOP_SESSION_ID, XDG_CURRENT_DESKTOP,
+     GNOME_KEYRING_CONTROL, DISPLAY, SSH_AGENT_PID, LANG, no_proxy,
+     XDG_SESSION_PATH, XAUTHORITY, SESSION_MANAGER, SHLVL,
+     MANDATORY_PATH, COMPIZ_CONFIG_PROFILE, WINDOWID, EDITOR,
+     GPG_AGENT_INFO, SSH_AUTH_SOCK, GDMSESSION, GNOME_KEYRING_PID,
+     XDG_SEAT_PATH, XDG_CONFIG_DIRS, LESSOPEN, DBUS_SESSION_BUS_ADDRESS,
+     _, XDG_SESSION_COOKIE, DESKTOP_SESSION, LESSCLOSE, DEFAULTS_PATH,
+     UBUNTU_MENUPROXY, OLDPWD, XDG_DATA_DIRS, COLORTERM, LS_COLORS
+                </literallayout>
+                The majority of this output is specific to environment variables
+                that are not directly relevant to BitBake.
+                However, the very first message regarding the
+                <filename>BBPATH</filename> variable and the
+                <filename>conf/bblayers.conf</filename> file
+                is relevant.</para>
+                <para>
+                When you run BitBake, it begins looking for metadata files.
+                The
+                <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
+                variable is what tells BitBake where to look for those files.
+                <filename>BBPATH</filename> is not set and you need to set it.
+                Without <filename>BBPATH</filename>, Bitbake cannot
+                find any configuration files (<filename>.conf</filename>)
+                or recipe files (<filename>.bb</filename>) at all.
+                BitBake also cannot find the <filename>bitbake.conf</filename>
+                file.
+                </para></listitem>
+            <listitem><para><emphasis>Setting <filename>BBPATH</filename>:</emphasis>
+                For this example, you can set <filename>BBPATH</filename>
+                in the same manner that you set <filename>PATH</filename>
+                earlier in the appendix.
+                You should realize, though, that it is much more flexible to set the
+                <filename>BBPATH</filename> variable up in a configuration
+                file for each project.</para>
+                <para>From your shell, enter the following commands to set and
+                export the <filename>BBPATH</filename> variable:
+                <literallayout class='monospaced'>
+     $ BBPATH="<replaceable>projectdirectory</replaceable>"
+     $ export BBPATH
+                </literallayout>
+                Use your actual project directory in the command.
+                BitBake uses that directory to find the metadata it needs for
+                your project.
+                <note>
+                    When specifying your project directory, do not use the
+                    tilde ("~") character as BitBake does not expand that character
+                    as the shell would.
+                </note>
+                </para></listitem>
+            <listitem><para><emphasis>Run Bitbake:</emphasis>
+                Now that you have <filename>BBPATH</filename> defined, run
+                the <filename>bitbake</filename> command again:
+                <literallayout class='monospaced'>
+     $ bitbake
+     ERROR: Traceback (most recent call last):
+       File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 163, in wrapped
+         return func(fn, *args)
+       File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 173, in parse_config_file
+         return bb.parse.handle(fn, data, include)
+       File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line 99, in handle
+         return h['handle'](fn, data, include)
+       File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 120, in handle
+         abs_fn = resolve_file(fn, data)
+       File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line 117, in resolve_file
+         raise IOError("file %s not found in %s" % (fn, bbpath))
+     IOError: file conf/bitbake.conf not found in /home/scott-lenovo/hello
+
+     ERROR: Unable to parse conf/bitbake.conf: file conf/bitbake.conf not found in /home/scott-lenovo/hello
+                </literallayout>
+                This sample output shows that BitBake could not find the
+                <filename>conf/bitbake.conf</filename> file in the project
+                directory.
+                This file is the first thing BitBake must find in order
+                to build a target.
+                And, since the project directory for this example is
+                empty, you need to provide a <filename>conf/bitbake.conf</filename>
+                file.
+                </para></listitem>
+            <listitem><para><emphasis>Creating <filename>conf/bitbake.conf</filename>:</emphasis>
+                The <filename>conf/bitbake.conf</filename> includes a number of
+                configuration variables BitBake uses for metadata and recipe
+                files.
+                For this example, you need to create the file in your project directory
+                and define some key BitBake variables.
+                For more information on the <filename>bitbake.conf</filename>,
+                see
+                <ulink url='http://git.openembedded.org/bitbake/tree/conf/bitbake.conf'></ulink>.
+                </para>
+                <para>Use the following commands to create the <filename>conf</filename>
+                directory in the project directory:
+                <literallayout class='monospaced'>
+     $ mkdir conf
+                </literallayout>
+                From within the <filename>conf</filename> directory, use
+                some editor to create the <filename>bitbake.conf</filename>
+                so that it contains the following:
+                <literallayout class='monospaced'>
+     TMPDIR  = "${<link linkend='var-TOPDIR'>TOPDIR</link>}/tmp"
+     <link linkend='var-CACHE'>CACHE</link>   = "${TMPDIR}/cache"
+     <link linkend='var-STAMP'>STAMP</link>   = "${TMPDIR}/stamps"
+     <link linkend='var-T'>T</link>       = "${TMPDIR}/work"
+     <link linkend='var-B'>B</link>       = "${TMPDIR}"
+                </literallayout>
+                The <filename>TMPDIR</filename> variable establishes a directory
+                that BitBake uses for build output and intermediate files (other
+                than the cached information used by the
+                <link linkend='setscene'>Setscene</link> process.
+                Here, the <filename>TMPDIR</filename> directory is set to
+                <filename>hello/tmp</filename>.
+                <note><title>Tip</title>
+                    You can always safely delete the <filename>tmp</filename>
+                    directory in order to rebuild a BitBake target.
+                    The build process creates the directory for you
+                    when you run BitBake.
+                </note></para>
+                <para>For information about each of the other variables defined in this
+                example, click on the links to take you to the definitions in
+                the glossary.
+                </para></listitem>
+            <listitem><para><emphasis>Run Bitbake:</emphasis>
+                After making sure that the <filename>conf/bitbake.conf</filename>
+                file exists, you can run the <filename>bitbake</filename>
+                command again:
+                <literallayout class='monospaced'>
+$ bitbake
+ERROR: Traceback (most recent call last):
+  File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 163, in wrapped
+    return func(fn, *args)
+  File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 177, in _inherit
+    bb.parse.BBHandler.inherit(bbclass, "configuration INHERITs", 0, data)
+  File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 92, in inherit
+    include(fn, file, lineno, d, "inherit")
+  File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 100, in include
+    raise ParseError("Could not %(error_out)s file %(fn)s" % vars(), oldfn, lineno)
+ParseError: ParseError in configuration INHERITs: Could not inherit file classes/base.bbclass
+
+ERROR: Unable to parse base: ParseError in configuration INHERITs: Could not inherit file classes/base.bbclass
+                </literallayout>
+                In the sample output, BitBake could not find the
+                <filename>classes/base.bbclass</filename> file.
+                You need to create that file next.
+                </para></listitem>
+            <listitem><para><emphasis>Creating <filename>classes/base.bbclass</filename>:</emphasis>
+                BitBake uses class files to provide common code and functionality.
+                The minimally required class for BitBake is the
+                <filename>classes/base.bbclass</filename> file.
+                The <filename>base</filename> class is implicitly inherited by
+                every recipe.
+                BitBake looks for the class in the <filename>classes</filename>
+                directory of the project (i.e <filename>hello/classes</filename>
+                in this example).
+                </para>
+                <para>Create the <filename>classes</filename> directory as follows:
+                <literallayout class='monospaced'>
+     $ cd $HOME/hello
+     $ mkdir classes
+                </literallayout>
+                Move to the <filename>classes</filename> directory and then
+                create the <filename>base.bbclass</filename> file by inserting
+                this single line:
+                <literallayout class='monospaced'>
+     addtask build
+                </literallayout>
+                The minimal task that BitBake runs is the
+                <filename>do_build</filename> task.
+                This is all the example needs in order to build the project.
+                Of course, the <filename>base.bbclass</filename> can have much
+                more depending on which build environments BitBake is
+                supporting.
+                </para></listitem>
+            <listitem><para><emphasis>Run Bitbake:</emphasis>
+                After making sure that the <filename>classes/base.bbclass</filename>
+                file exists, you can run the <filename>bitbake</filename>
+                command again:
+                <literallayout class='monospaced'>
+     $ bitbake
+     Nothing to do.  Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.
+                </literallayout>
+                BitBake is finally reporting no errors.
+                However, you can see that it really does not have anything
+                to do.
+                You need to create a recipe that gives BitBake something to do.
+                </para></listitem>
+            <listitem><para><emphasis>Creating a Layer:</emphasis>
+                While it is not really necessary for such a small example,
+                it is good practice to create a layer in which to keep your
+                code separate from the general metadata used by BitBake.
+                Thus, this example creates and uses a layer called "mylayer".
+                <note>
+                    You can find additional information on layers at
+                    <ulink url='http://www.yoctoproject.org/docs/2.3/bitbake-user-manual/bitbake-user-manual.html#layers'></ulink>.
+                </note>
+                </para>
+                <para>Minimally, you need a recipe file and a layer configuration
+                file in your layer.
+                The configuration file needs to be in the <filename>conf</filename>
+                directory inside the layer.
+                Use these commands to set up the layer and the <filename>conf</filename>
+                directory:
+                <literallayout class='monospaced'>
+     $ cd $HOME
+     $ mkdir mylayer
+     $ cd mylayer
+     $ mkdir conf
+                </literallayout>
+                Move to the <filename>conf</filename> directory and create a
+                <filename>layer.conf</filename> file that has the following:
+                <literallayout class='monospaced'>
+     BBPATH .= ":${<link linkend='var-LAYERDIR'>LAYERDIR</link>}"
+
+     <link linkend='var-BBFILES'>BBFILES</link> += "${LAYERDIR}/*.bb"
+
+     <link linkend='var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</link> += "mylayer"
+     <link linkend='var-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer</link> := "^${LAYERDIR_RE}/"
+                </literallayout>
+                For information on these variables, click the links
+                to go to the definitions in the glossary.</para>
+                <para>You need to create the recipe file next.
+                Inside your layer at the top-level, use an editor and create
+                a recipe file named <filename>printhello.bb</filename> that
+                has the following:
+                <literallayout class='monospaced'>
+     <link linkend='var-DESCRIPTION'>DESCRIPTION</link> = "Prints Hello World"
+     <link linkend='var-PN'>PN</link> = 'printhello'
+     <link linkend='var-PV'>PV</link> = '1'
+
+     python do_build() {
+        bb.plain("********************");
+        bb.plain("*                  *");
+        bb.plain("*  Hello, World!   *");
+        bb.plain("*                  *");
+        bb.plain("********************");
+     }
+                </literallayout>
+                The recipe file simply provides a description of the
+                recipe, the name, version, and the <filename>do_build</filename>
+                task, which prints out "Hello World" to the console.
+                For more information on these variables, follow the links
+                to the glossary.
+                </para></listitem>
+            <listitem><para><emphasis>Run Bitbake With a Target:</emphasis>
+                Now that a BitBake target exists, run the command and provide
+                that target:
+                <literallayout class='monospaced'>
+     $ cd $HOME/hello
+     $ bitbake printhello
+     ERROR: no recipe files to build, check your BBPATH and BBFILES?
+
+     Summary: There was 1 ERROR message shown, returning a non-zero exit code.
+                </literallayout>
+                We have created the layer with the recipe and the layer
+                configuration file but it still seems that BitBake cannot
+                find the recipe.
+                BitBake needs a <filename>conf/bblayers.conf</filename> that
+                lists the layers for the project.
+                Without this file, BitBake cannot find the recipe.
+                </para></listitem>
+            <listitem><para><emphasis>Creating <filename>conf/bblayers.conf</filename>:</emphasis>
+                BitBake uses the <filename>conf/bblayers.conf</filename> file
+                to locate layers needed for the project.
+                This file must reside in the <filename>conf</filename> directory
+                of the project (i.e. <filename>hello/conf</filename> for this
+                example).</para>
+                <para>Set your working directory to the <filename>hello/conf</filename>
+                directory and then create the <filename>bblayers.conf</filename>
+                file so that it contains the following:
+                <literallayout class='monospaced'>
+     BBLAYERS ?= " \
+       /home/&lt;you&gt;/mylayer \
+       "
+                </literallayout>
+                You need to provide your own information for
+                <filename>you</filename> in the file.
+                </para></listitem>
+            <listitem><para><emphasis>Run Bitbake With a Target:</emphasis>
+                Now that you have supplied the <filename>bblayers.conf</filename>
+                file, run the <filename>bitbake</filename> command and provide
+                the target:
+                <literallayout class='monospaced'>
+     $ bitbake printhello
+     Parsing recipes: 100% |##################################################################################|
+     Time: 00:00:00
+     Parsing of 1 .bb files complete (0 cached, 1 parsed). 1 targets, 0 skipped, 0 masked, 0 errors.
+     NOTE: Resolving any missing task queue dependencies
+     NOTE: Preparing RunQueue
+     NOTE: Executing RunQueue Tasks
+     ********************
+     *                  *
+     *  Hello, World!   *
+     *                  *
+     ********************
+     NOTE: Tasks Summary: Attempted 1 tasks of which 0 didn't need to be rerun and all succeeded.
+                </literallayout>
+                BitBake finds the <filename>printhello</filename> recipe and
+                successfully runs the task.
+                <note>
+                    After the first execution, re-running
+                    <filename>bitbake printhello</filename> again will not
+                    result in a BitBake run that prints the same console
+                    output.
+                    The reason for this is that the first time the
+                    <filename>printhello.bb</filename> recipe's
+                    <filename>do_build</filename> task executes
+                    successfully, BitBake writes a stamp file for the task.
+                    Thus, the next time you attempt to run the task
+                    using that same <filename>bitbake</filename> command,
+                    BitBake notices the stamp and therefore determines
+                    that the task does not need to be re-run.
+                    If you delete the <filename>tmp</filename> directory
+                    or run <filename>bitbake -c clean printhello</filename>
+                    and then re-run the build, the "Hello, World!" message will
+                    be printed again.
+                </note>
+                </para></listitem>
+        </orderedlist>
+    </section>
+</appendix>

bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst

@@ -1,651 +0,0 @@
-.. SPDX-License-Identifier: CC-BY-2.5
-
-========
-Overview
-========
-
-|
-
-Welcome to the BitBake User Manual. This manual provides information on
-the BitBake tool. The information attempts to be as independent as
-possible regarding systems that use BitBake, such as OpenEmbedded and
-the Yocto Project. In some cases, scenarios or examples within the
-context of a build system are used in the manual to help with
-understanding. For these cases, the manual clearly states the context.
-
-.. _intro:
-
-Introduction
-============
-
-Fundamentally, BitBake is a generic task execution engine that allows
-shell and Python tasks to be run efficiently and in parallel while
-working within complex inter-task dependency constraints. One of
-BitBake's main users, OpenEmbedded, takes this core and builds embedded
-Linux software stacks using a task-oriented approach.
-
-Conceptually, BitBake is similar to GNU Make in some regards but has
-significant differences:
-
--  BitBake executes tasks according to provided metadata that builds up
-   the tasks. Metadata is stored in recipe (``.bb``) and related recipe
-   "append" (``.bbappend``) files, configuration (``.conf``) and
-   underlying include (``.inc``) files, and in class (``.bbclass``)
-   files. The metadata provides BitBake with instructions on what tasks
-   to run and the dependencies between those tasks.
-
--  BitBake includes a fetcher library for obtaining source code from
-   various places such as local files, source control systems, or
-   websites.
-
--  The instructions for each unit to be built (e.g. a piece of software)
-   are known as "recipe" files and contain all the information about the
-   unit (dependencies, source file locations, checksums, description and
-   so on).
-
--  BitBake includes a client/server abstraction and can be used from a
-   command line or used as a service over XML-RPC and has several
-   different user interfaces.
-
-History and Goals
-=================
-
-BitBake was originally a part of the OpenEmbedded project. It was
-inspired by the Portage package management system used by the Gentoo
-Linux distribution. On December 7, 2004, OpenEmbedded project team
-member Chris Larson split the project into two distinct pieces:
-
--  BitBake, a generic task executor
-
--  OpenEmbedded, a metadata set utilized by BitBake
-
-Today, BitBake is the primary basis of the
-`OpenEmbedded <http://www.openembedded.org/>`__ project, which is being
-used to build and maintain Linux distributions such as the `Angstrom
-Distribution <http://www.angstrom-distribution.org/>`__, and which is
-also being used as the build tool for Linux projects such as the `Yocto
-Project <http://www.yoctoproject.org>`__.
-
-Prior to BitBake, no other build tool adequately met the needs of an
-aspiring embedded Linux distribution. All of the build systems used by
-traditional desktop Linux distributions lacked important functionality,
-and none of the ad hoc Buildroot-based systems, prevalent in the
-embedded space, were scalable or maintainable.
-
-Some important original goals for BitBake were:
-
--  Handle cross-compilation.
-
--  Handle inter-package dependencies (build time on target architecture,
-   build time on native architecture, and runtime).
-
--  Support running any number of tasks within a given package,
-   including, but not limited to, fetching upstream sources, unpacking
-   them, patching them, configuring them, and so forth.
-
--  Be Linux distribution agnostic for both build and target systems.
-
--  Be architecture agnostic.
-
--  Support multiple build and target operating systems (e.g. Cygwin, the
-   BSDs, and so forth).
-
--  Be self-contained, rather than tightly integrated into the build
-   machine's root filesystem.
-
--  Handle conditional metadata on the target architecture, operating
-   system, distribution, and machine.
-
--  Be easy to use the tools to supply local metadata and packages
-   against which to operate.
-
--  Be easy to use BitBake to collaborate between multiple projects for
-   their builds.
-
--  Provide an inheritance mechanism to share common metadata between
-   many packages.
-
-Over time it became apparent that some further requirements were
-necessary:
-
--  Handle variants of a base recipe (e.g. native, sdk, and multilib).
-
--  Split metadata into layers and allow layers to enhance or override
-   other layers.
-
--  Allow representation of a given set of input variables to a task as a
-   checksum. Based on that checksum, allow acceleration of builds with
-   prebuilt components.
-
-BitBake satisfies all the original requirements and many more with
-extensions being made to the basic functionality to reflect the
-additional requirements. Flexibility and power have always been the
-priorities. BitBake is highly extensible and supports embedded Python
-code and execution of any arbitrary tasks.
-
-.. _Concepts:
-
-Concepts
-========
-
-BitBake is a program written in the Python language. At the highest
-level, BitBake interprets metadata, decides what tasks are required to
-run, and executes those tasks. Similar to GNU Make, BitBake controls how
-software is built. GNU Make achieves its control through "makefiles",
-while BitBake uses "recipes".
-
-BitBake extends the capabilities of a simple tool like GNU Make by
-allowing for the definition of much more complex tasks, such as
-assembling entire embedded Linux distributions.
-
-The remainder of this section introduces several concepts that should be
-understood in order to better leverage the power of BitBake.
-
-Recipes
--------
-
-BitBake Recipes, which are denoted by the file extension ``.bb``, are
-the most basic metadata files. These recipe files provide BitBake with
-the following:
-
--  Descriptive information about the package (author, homepage, license,
-   and so on)
-
--  The version of the recipe
-
--  Existing dependencies (both build and runtime dependencies)
-
--  Where the source code resides and how to fetch it
-
--  Whether the source code requires any patches, where to find them, and
-   how to apply them
-
--  How to configure and compile the source code
-
--  How to assemble the generated artifacts into one or more installable
-   packages
-
--  Where on the target machine to install the package or packages
-   created
-
-Within the context of BitBake, or any project utilizing BitBake as its
-build system, files with the ``.bb`` extension are referred to as
-recipes.
-
-.. note::
-
-   The term "package" is also commonly used to describe recipes.
-   However, since the same word is used to describe packaged output from
-   a project, it is best to maintain a single descriptive term -
-   "recipes". Put another way, a single "recipe" file is quite capable
-   of generating a number of related but separately installable
-   "packages". In fact, that ability is fairly common.
-
-Configuration Files
--------------------
-
-Configuration files, which are denoted by the ``.conf`` extension,
-define various configuration variables that govern the project's build
-process. These files fall into several areas that define machine
-configuration, distribution configuration, possible compiler tuning,
-general common configuration, and user configuration. The main
-configuration file is the sample ``bitbake.conf`` file, which is located
-within the BitBake source tree ``conf`` directory.
-
-Classes
--------
-
-Class files, which are denoted by the ``.bbclass`` extension, contain
-information that is useful to share between metadata files. The BitBake
-source tree currently comes with one class metadata file called
-``base.bbclass``. You can find this file in the ``classes`` directory.
-The ``base.bbclass`` class files is special since it is always included
-automatically for all recipes and classes. This class contains
-definitions for standard basic tasks such as fetching, unpacking,
-configuring (empty by default), compiling (runs any Makefile present),
-installing (empty by default) and packaging (empty by default). These
-tasks are often overridden or extended by other classes added during the
-project development process.
-
-Layers
-------
-
-Layers allow you to isolate different types of customizations from each
-other. While you might find it tempting to keep everything in one layer
-when working on a single project, the more modular your metadata, the
-easier it is to cope with future changes.
-
-To illustrate how you can use layers to keep things modular, consider
-customizations you might make to support a specific target machine.
-These types of customizations typically reside in a special layer,
-rather than a general layer, called a Board Support Package (BSP) layer.
-Furthermore, the machine customizations should be isolated from recipes
-and metadata that support a new GUI environment, for example. This
-situation gives you a couple of layers: one for the machine
-configurations and one for the GUI environment. It is important to
-understand, however, that the BSP layer can still make machine-specific
-additions to recipes within the GUI environment layer without polluting
-the GUI layer itself with those machine-specific changes. You can
-accomplish this through a recipe that is a BitBake append
-(``.bbappend``) file.
-
-.. _append-bbappend-files:
-
-Append Files
-------------
-
-Append files, which are files that have the ``.bbappend`` file
-extension, extend or override information in an existing recipe file.
-
-BitBake expects every append file to have a corresponding recipe file.
-Furthermore, the append file and corresponding recipe file must use the
-same root filename. The filenames can differ only in the file type
-suffix used (e.g. ``formfactor_0.0.bb`` and
-``formfactor_0.0.bbappend``).
-
-Information in append files extends or overrides the information in the
-underlying, similarly-named recipe files.
-
-When you name an append file, you can use the "``%``" wildcard character
-to allow for matching recipe names. For example, suppose you have an
-append file named as follows: ::
-
-  busybox_1.21.%.bbappend
-
-That append file
-would match any ``busybox_1.21.``\ x\ ``.bb`` version of the recipe. So,
-the append file would match the following recipe names: ::
-
-  busybox_1.21.1.bb
-  busybox_1.21.2.bb
-  busybox_1.21.3.bb
-
-.. note::
-
-   The use of the " % " character is limited in that it only works directly in
-   front of the .bbappend portion of the append file's name. You cannot use the
-   wildcard character in any other location of the name.
-
-If the ``busybox`` recipe was updated to ``busybox_1.3.0.bb``, the
-append name would not match. However, if you named the append file
-``busybox_1.%.bbappend``, then you would have a match.
-
-In the most general case, you could name the append file something as
-simple as ``busybox_%.bbappend`` to be entirely version independent.
-
-Obtaining BitBake
-=================
-
-You can obtain BitBake several different ways:
-
--  **Cloning BitBake:** Using Git to clone the BitBake source code
-   repository is the recommended method for obtaining BitBake. Cloning
-   the repository makes it easy to get bug fixes and have access to
-   stable branches and the master branch. Once you have cloned BitBake,
-   you should use the latest stable branch for development since the
-   master branch is for BitBake development and might contain less
-   stable changes.
-
-   You usually need a version of BitBake that matches the metadata you
-   are using. The metadata is generally backwards compatible but not
-   forward compatible.
-
-   Here is an example that clones the BitBake repository: ::
-
-     $ git clone git://git.openembedded.org/bitbake
-
-   This command clones the BitBake
-   Git repository into a directory called ``bitbake``. Alternatively,
-   you can designate a directory after the ``git clone`` command if you
-   want to call the new directory something other than ``bitbake``. Here
-   is an example that names the directory ``bbdev``: ::
-
-     $ git clone git://git.openembedded.org/bitbake bbdev
-
--  **Installation using your Distribution Package Management System:**
-   This method is not recommended because the BitBake version that is
-   provided by your distribution, in most cases, is several releases
-   behind a snapshot of the BitBake repository.
-
--  **Taking a snapshot of BitBake:** Downloading a snapshot of BitBake
-   from the source code repository gives you access to a known branch or
-   release of BitBake.
-
-      .. note::
-
-         Cloning the Git repository, as described earlier, is the preferred
-         method for getting BitBake. Cloning the repository makes it easier
-         to update as patches are added to the stable branches.
-
-   The following example downloads a snapshot of BitBake version 1.17.0: ::
-
-     $ wget http://git.openembedded.org/bitbake/snapshot/bitbake-1.17.0.tar.gz
-     $ tar zxpvf bitbake-1.17.0.tar.gz
-
-   After extraction of the tarball using
-   the tar utility, you have a directory entitled ``bitbake-1.17.0``.
-
--  **Using the BitBake that Comes With Your Build Checkout:** A final
-   possibility for getting a copy of BitBake is that it already comes
-   with your checkout of a larger BitBake-based build system, such as
-   Poky. Rather than manually checking out individual layers and gluing
-   them together yourself, you can check out an entire build system. The
-   checkout will already include a version of BitBake that has been
-   thoroughly tested for compatibility with the other components. For
-   information on how to check out a particular BitBake-based build
-   system, consult that build system's supporting documentation.
-
-.. _bitbake-user-manual-command:
-
-The BitBake Command
-===================
-
-The ``bitbake`` command is the primary interface to the BitBake tool.
-This section presents the BitBake command syntax and provides several
-execution examples.
-
-Usage and syntax
-----------------
-
-Following is the usage and syntax for BitBake: ::
-
-   $ bitbake -h
-   Usage: bitbake [options] [recipename/target recipe:do_task ...]
-
-       Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
-       It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
-       will provide the layer, BBFILES and other configuration information.
-
-   Options:
-     --version             show program's version number and exit
-     -h, --help            show this help message and exit
-     -b BUILDFILE, --buildfile=BUILDFILE
-                           Execute tasks from a specific .bb recipe directly.
-                           WARNING: Does not handle any dependencies from other
-                           recipes.
-     -k, --continue        Continue as much as possible after an error. While the
-                           target that failed and anything depending on it cannot
-                           be built, as much as possible will be built before
-                           stopping.
-     -f, --force           Force the specified targets/task to run (invalidating
-                           any existing stamp file).
-     -c CMD, --cmd=CMD     Specify the task to execute. The exact options
-                           available depend on the metadata. Some examples might
-                           be 'compile' or 'populate_sysroot' or 'listtasks' may
-                           give a list of the tasks available.
-     -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP
-                           Invalidate the stamp for the specified task such as
-                           'compile' and then run the default task for the
-                           specified target(s).
-     -r PREFILE, --read=PREFILE
-                           Read the specified file before bitbake.conf.
-     -R POSTFILE, --postread=POSTFILE
-                           Read the specified file after bitbake.conf.
-     -v, --verbose         Enable tracing of shell tasks (with 'set -x'). Also
-                           print bb.note(...) messages to stdout (in addition to
-                           writing them to ${T}/log.do_&lt;task&gt;).
-     -D, --debug           Increase the debug level. You can specify this more
-                           than once. -D sets the debug level to 1, where only
-                           bb.debug(1, ...) messages are printed to stdout; -DD
-                           sets the debug level to 2, where both bb.debug(1, ...)
-                           and bb.debug(2, ...) messages are printed; etc.
-                           Without -D, no debug messages are printed. Note that
-                           -D only affects output to stdout. All debug messages
-                           are written to ${T}/log.do_taskname, regardless of the
-                           debug level.
-     -q, --quiet           Output less log message data to the terminal. You can
-                           specify this more than once.
-     -n, --dry-run         Don't execute, just go through the motions.
-     -S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER
-                           Dump out the signature construction information, with
-                           no task execution. The SIGNATURE_HANDLER parameter is
-                           passed to the handler. Two common values are none and
-                           printdiff but the handler may define more/less. none
-                           means only dump the signature, printdiff means compare
-                           the dumped signature with the cached one.
-     -p, --parse-only      Quit after parsing the BB recipes.
-     -s, --show-versions   Show current and preferred versions of all recipes.
-     -e, --environment     Show the global or per-recipe environment complete
-                           with information about where variables were
-                           set/changed.
-     -g, --graphviz        Save dependency tree information for the specified
-                           targets in the dot syntax.
-     -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
-                           Assume these dependencies don't exist and are already
-                           provided (equivalent to ASSUME_PROVIDED). Useful to
-                           make dependency graphs more appealing
-     -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
-                           Show debug logging for the specified logging domains
-     -P, --profile         Profile the command and save reports.
-     -u UI, --ui=UI        The user interface to use (knotty, ncurses or taskexp
-                           - default knotty).
-     --token=XMLRPCTOKEN   Specify the connection token to be used when
-                           connecting to a remote server.
-     --revisions-changed   Set the exit code depending on whether upstream
-                           floating revisions have changed or not.
-     --server-only         Run bitbake without a UI, only starting a server
-                           (cooker) process.
-     -B BIND, --bind=BIND  The name/address for the bitbake xmlrpc server to bind
-                           to.
-     -T SERVER_TIMEOUT, --idle-timeout=SERVER_TIMEOUT
-                           Set timeout to unload bitbake server due to
-                           inactivity, set to -1 means no unload, default:
-                           Environment variable BB_SERVER_TIMEOUT.
-     --no-setscene         Do not run any setscene tasks. sstate will be ignored
-                           and everything needed, built.
-     --setscene-only       Only run setscene tasks, don't run any real tasks.
-     --remote-server=REMOTE_SERVER
-                           Connect to the specified server.
-     -m, --kill-server     Terminate any running bitbake server.
-     --observe-only        Connect to a server as an observing-only client.
-     --status-only         Check the status of the remote bitbake server.
-     -w WRITEEVENTLOG, --write-log=WRITEEVENTLOG
-                           Writes the event log of the build to a bitbake event
-                           json file. Use '' (empty string) to assign the name
-                           automatically.
-     --runall=RUNALL       Run the specified task for any recipe in the taskgraph
-                           of the specified target (even if it wouldn't otherwise
-                           have run).
-     --runonly=RUNONLY     Run only the specified task within the taskgraph of
-                           the specified targets (and any task dependencies those
-                           tasks may have).
-
-.. _bitbake-examples:
-
-Examples
---------
-
-This section presents some examples showing how to use BitBake.
-
-.. _example-executing-a-task-against-a-single-recipe:
-
-Executing a Task Against a Single Recipe
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Executing tasks for a single recipe file is relatively simple. You
-specify the file in question, and BitBake parses it and executes the
-specified task. If you do not specify a task, BitBake executes the
-default task, which is "build". BitBake obeys inter-task dependencies
-when doing so.
-
-The following command runs the build task, which is the default task, on
-the ``foo_1.0.bb`` recipe file: ::
-
-  $ bitbake -b foo_1.0.bb
-
-The following command runs the clean task on the ``foo.bb`` recipe file: ::
-
-  $ bitbake -b foo.bb -c clean
-
-.. note::
-
-   The "-b" option explicitly does not handle recipe dependencies. Other
-   than for debugging purposes, it is instead recommended that you use
-   the syntax presented in the next section.
-
-Executing Tasks Against a Set of Recipe Files
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-There are a number of additional complexities introduced when one wants
-to manage multiple ``.bb`` files. Clearly there needs to be a way to
-tell BitBake what files are available and, of those, which you want to
-execute. There also needs to be a way for each recipe to express its
-dependencies, both for build-time and runtime. There must be a way for
-you to express recipe preferences when multiple recipes provide the same
-functionality, or when there are multiple versions of a recipe.
-
-The ``bitbake`` command, when not using "--buildfile" or "-b" only
-accepts a "PROVIDES". You cannot provide anything else. By default, a
-recipe file generally "PROVIDES" its "packagename" as shown in the
-following example: ::
-
-  $ bitbake foo
-
-This next example "PROVIDES" the
-package name and also uses the "-c" option to tell BitBake to just
-execute the ``do_clean`` task: ::
-
-  $ bitbake -c clean foo
-
-Executing a List of Task and Recipe Combinations
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The BitBake command line supports specifying different tasks for
-individual targets when you specify multiple targets. For example,
-suppose you had two targets (or recipes) ``myfirstrecipe`` and
-``mysecondrecipe`` and you needed BitBake to run ``taskA`` for the first
-recipe and ``taskB`` for the second recipe: ::
-
-  $ bitbake myfirstrecipe:do_taskA mysecondrecipe:do_taskB
-
-Generating Dependency Graphs
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-BitBake is able to generate dependency graphs using the ``dot`` syntax.
-You can convert these graphs into images using the ``dot`` tool from
-`Graphviz <http://www.graphviz.org>`__.
-
-When you generate a dependency graph, BitBake writes two files to the
-current working directory:
-
--  ``task-depends.dot``: Shows dependencies between tasks. These
-   dependencies match BitBake's internal task execution list.
-
--  ``pn-buildlist``: Shows a simple list of targets that are to be
-   built.
-
-To stop depending on common depends, use the "-I" depend option and
-BitBake omits them from the graph. Leaving this information out can
-produce more readable graphs. This way, you can remove from the graph
-``DEPENDS`` from inherited classes such as ``base.bbclass``.
-
-Here are two examples that create dependency graphs. The second example
-omits depends common in OpenEmbedded from the graph: ::
-
-  $ bitbake -g foo
-
-  $ bitbake -g -I virtual/kernel -I eglibc foo
-
-Executing a Multiple Configuration Build
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-BitBake is able to build multiple images or packages using a single
-command where the different targets require different configurations
-(multiple configuration builds). Each target, in this scenario, is
-referred to as a "multiconfig".
-
-To accomplish a multiple configuration build, you must define each
-target's configuration separately using a parallel configuration file in
-the build directory. The location for these multiconfig configuration
-files is specific. They must reside in the current build directory in a
-sub-directory of ``conf`` named ``multiconfig``. Following is an example
-for two separate targets:
-
-.. image:: figures/bb_multiconfig_files.png
-   :align: center
-
-The reason for this required file hierarchy is because the ``BBPATH``
-variable is not constructed until the layers are parsed. Consequently,
-using the configuration file as a pre-configuration file is not possible
-unless it is located in the current working directory.
-
-Minimally, each configuration file must define the machine and the
-temporary directory BitBake uses for the build. Suggested practice
-dictates that you do not overlap the temporary directories used during
-the builds.
-
-Aside from separate configuration files for each target, you must also
-enable BitBake to perform multiple configuration builds. Enabling is
-accomplished by setting the
-:term:`BBMULTICONFIG` variable in the
-``local.conf`` configuration file. As an example, suppose you had
-configuration files for ``target1`` and ``target2`` defined in the build
-directory. The following statement in the ``local.conf`` file both
-enables BitBake to perform multiple configuration builds and specifies
-the two extra multiconfigs: ::
-
-  BBMULTICONFIG = "target1 target2"
-
-Once the target configuration files are in place and BitBake has been
-enabled to perform multiple configuration builds, use the following
-command form to start the builds: ::
-
-  $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
-
-Here is an example for two extra multiconfigs: ``target1`` and ``target2``: ::
-
-  $ bitbake mc::target mc:target1:target mc:target2:target
-
-.. _bb-enabling-multiple-configuration-build-dependencies:
-
-Enabling Multiple Configuration Build Dependencies
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Sometimes dependencies can exist between targets (multiconfigs) in a
-multiple configuration build. For example, suppose that in order to
-build an image for a particular architecture, the root filesystem of
-another build for a different architecture needs to exist. In other
-words, the image for the first multiconfig depends on the root
-filesystem of the second multiconfig. This dependency is essentially
-that the task in the recipe that builds one multiconfig is dependent on
-the completion of the task in the recipe that builds another
-multiconfig.
-
-To enable dependencies in a multiple configuration build, you must
-declare the dependencies in the recipe using the following statement
-form: ::
-
-  task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
-
-To better show how to use this statement, consider an example with two
-multiconfigs: ``target1`` and ``target2``: ::
-
-  image_task[mcdepends] = "mc:target1:target2:image2:rootfs_task"
-
-In this example, the
-``from_multiconfig`` is "target1" and the ``to_multiconfig`` is "target2". The
-task on which the image whose recipe contains image_task depends on the
-completion of the rootfs_task used to build out image2, which is
-associated with the "target2" multiconfig.
-
-Once you set up this dependency, you can build the "target1" multiconfig
-using a BitBake command as follows: ::
-
-  $ bitbake mc:target1:image1
-
-This command executes all the tasks needed to create ``image1`` for the "target1"
-multiconfig. Because of the dependency, BitBake also executes through
-the ``rootfs_task`` for the "target2" multiconfig build.
-
-Having a recipe depend on the root filesystem of another build might not
-seem that useful. Consider this change to the statement in the image1
-recipe: ::
-
-  image_task[mcdepends] = "mc:target1:target2:image2:image_task"
-
-In this case, BitBake must create ``image2`` for the "target2" build since
-the "target1" build depends on it.
-
-Because "target1" and "target2" are enabled for multiple configuration
-builds and have separate configuration files, BitBake places the
-artifacts for each build in the respective temporary build directories.

bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml

@@ -0,0 +1,721 @@
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+    "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+
+<chapter id="bitbake-user-manual-intro">
+    <title>Overview</title>
+
+    <para>
+        Welcome to the BitBake User Manual.
+        This manual provides information on the BitBake tool.
+        The information attempts to be as independent as possible regarding
+        systems that use BitBake, such as OpenEmbedded and the
+        Yocto Project.
+        In some cases, scenarios or examples within the context of
+        a build system are used in the manual to help with understanding.
+        For these cases, the manual clearly states the context.
+    </para>
+
+    <section id="intro">
+        <title>Introduction</title>
+
+        <para>
+            Fundamentally, BitBake is a generic task execution
+            engine that allows shell and Python tasks to be run
+            efficiently and in parallel while working within
+            complex inter-task dependency constraints.
+            One of BitBake's main users, OpenEmbedded, takes this core
+            and builds embedded Linux software stacks using
+            a task-oriented approach.
+        </para>
+
+        <para>
+            Conceptually, BitBake is similar to GNU Make in
+            some regards but has significant differences:
+            <itemizedlist>
+                <listitem><para>
+                    BitBake executes tasks according to provided
+                    metadata that builds up the tasks.
+                    Metadata is stored in recipe (<filename>.bb</filename>)
+                    and related recipe "append" (<filename>.bbappend</filename>)
+                    files, configuration (<filename>.conf</filename>) and
+                    underlying include (<filename>.inc</filename>) files, and
+                    in class (<filename>.bbclass</filename>) files.
+                    The metadata provides
+                    BitBake with instructions on what tasks to run and
+                    the dependencies between those tasks.
+                    </para></listitem>
+                <listitem><para>
+                    BitBake includes a fetcher library for obtaining source
+                    code from various places such as local files, source control
+                    systems, or websites.
+                    </para></listitem>
+                <listitem><para>
+                    The instructions for each unit to be built (e.g. a piece
+                    of software) are known as "recipe" files and
+                    contain all the information about the unit
+                    (dependencies, source file locations, checksums, description
+                    and so on).
+                    </para></listitem>
+                <listitem><para>
+                    BitBake includes a client/server abstraction and can
+                    be used from a command line or used as a service over
+                    XML-RPC and has several different user interfaces.
+                    </para></listitem>
+            </itemizedlist>
+        </para>
+    </section>
+
+    <section id="history-and-goals">
+        <title>History and Goals</title>
+
+        <para>
+            BitBake was originally a part of the OpenEmbedded project.
+            It was inspired by the Portage package management system
+            used by the Gentoo Linux distribution.
+            On December 7, 2004, OpenEmbedded project team member
+            Chris Larson split the project into two distinct pieces:
+            <itemizedlist>
+                <listitem><para>BitBake, a generic task executor</para></listitem>
+                <listitem><para>OpenEmbedded, a metadata set utilized by
+                    BitBake</para></listitem>
+            </itemizedlist>
+            Today, BitBake is the primary basis of the
+            <ulink url="http://www.openembedded.org/">OpenEmbedded</ulink>
+            project, which is being used to build and maintain Linux
+            distributions such as the
+            <ulink url='http://www.angstrom-distribution.org/'>Angstrom Distribution</ulink>,
+            and which is also being used as the build tool for Linux projects
+            such as the
+            <ulink url='http://www.yoctoproject.org'>Yocto Project</ulink>.
+        </para>
+
+        <para>
+            Prior to BitBake, no other build tool adequately met the needs of
+            an aspiring embedded Linux distribution.
+            All of the build systems used by traditional desktop Linux
+            distributions lacked important functionality, and none of the
+            ad hoc Buildroot-based systems, prevalent in the
+            embedded space, were scalable or maintainable.
+        </para>
+
+        <para>
+            Some important original goals for BitBake were:
+            <itemizedlist>
+                <listitem><para>
+                    Handle cross-compilation.
+                    </para></listitem>
+                <listitem><para>
+                    Handle inter-package dependencies (build time on
+                    target architecture, build time on native
+                    architecture, and runtime).
+                    </para></listitem>
+                <listitem><para>
+                    Support running any number of tasks within a given
+                    package, including, but not limited to, fetching
+                    upstream sources, unpacking them, patching them,
+                    configuring them, and so forth.
+                    </para></listitem>
+                <listitem><para>
+                    Be Linux distribution agnostic for both build and
+                    target systems.
+                    </para></listitem>
+                <listitem><para>
+                    Be architecture agnostic.
+                    </para></listitem>
+                <listitem><para>
+                    Support multiple build and target operating systems
+                    (e.g. Cygwin, the BSDs, and so forth).
+                    </para></listitem>
+                <listitem><para>
+                    Be self contained, rather than tightly
+                    integrated into the build machine's root
+                    filesystem.
+                    </para></listitem>
+                <listitem><para>
+                    Handle conditional metadata on the target architecture,
+                    operating system, distribution, and machine.
+                    </para></listitem>
+                <listitem><para>
+                    Be easy to use the tools to supply local metadata and packages
+                    against which to operate.
+                    </para></listitem>
+                <listitem><para>
+                    Be easy to use BitBake to collaborate between multiple
+                    projects for their builds.
+                    </para></listitem>
+                <listitem><para>
+                    Provide an inheritance mechanism to share
+                    common metadata between many packages.
+                    </para></listitem>
+            </itemizedlist>
+            Over time it became apparent that some further requirements
+            were necessary:
+            <itemizedlist>
+                <listitem><para>
+                    Handle variants of a base recipe (e.g. native, sdk,
+                    and multilib).
+                    </para></listitem>
+                <listitem><para>
+                    Split metadata into layers and allow layers
+                    to enhance or override other layers.
+                    </para></listitem>
+                <listitem><para>
+                    Allow representation of a given set of input variables
+                    to a task as a checksum.
+                    Based on that checksum, allow acceleration of builds
+                    with prebuilt components.
+                    </para></listitem>
+            </itemizedlist>
+            BitBake satisfies all the original requirements and many more
+            with extensions being made to the basic functionality to
+            reflect the additional requirements.
+            Flexibility and power have always been the priorities.
+            BitBake is highly extensible and supports embedded Python code and
+            execution of any arbitrary tasks.
+        </para>
+    </section>
+
+    <section id="Concepts">
+        <title>Concepts</title>
+
+        <para>
+            BitBake is a program written in the Python language.
+            At the highest level, BitBake interprets metadata, decides
+            what tasks are required to run, and executes those tasks.
+            Similar to GNU Make, BitBake controls how software is
+            built.
+            GNU Make achieves its control through "makefiles", while
+            BitBake uses "recipes".
+        </para>
+
+        <para>
+            BitBake extends the capabilities of a simple
+            tool like GNU Make by allowing for the definition of much more
+            complex tasks, such as assembling entire embedded Linux
+            distributions.
+        </para>
+
+        <para>
+            The remainder of this section introduces several concepts
+            that should be understood in order to better leverage
+            the power of BitBake.
+        </para>
+
+        <section id='recipes'>
+            <title>Recipes</title>
+
+            <para>
+                BitBake Recipes, which are denoted by the file extension
+                <filename>.bb</filename>, are the most basic metadata files.
+                These recipe files provide BitBake with the following:
+                <itemizedlist>
+                    <listitem><para>Descriptive information about the
+                        package (author, homepage, license, and so on)</para></listitem>
+                    <listitem><para>The version of the recipe</para></listitem>
+                    <listitem><para>Existing dependencies (both build
+                        and runtime dependencies)</para></listitem>
+                    <listitem><para>Where the source code resides and
+                        how to fetch it</para></listitem>
+                    <listitem><para>Whether the source code requires
+                        any patches, where to find them, and how to apply
+                        them</para></listitem>
+                    <listitem><para>How to configure and compile the
+                        source code</para></listitem>
+                    <listitem><para>Where on the target machine to install the
+                        package or packages created</para></listitem>
+                </itemizedlist>
+            </para>
+
+            <para>
+                Within the context of BitBake, or any project utilizing BitBake
+                as its build system, files with the <filename>.bb</filename>
+                extension are referred to as recipes.
+                <note>
+                    The term "package" is also commonly used to describe recipes.
+                    However, since the same word is used to describe packaged
+                    output from a project, it is best to maintain a single
+                    descriptive term - "recipes".
+                    Put another way, a single "recipe" file is quite capable
+                    of generating a number of related but separately installable
+                    "packages".
+                    In fact, that ability is fairly common.
+                </note>
+            </para>
+        </section>
+
+        <section id='configuration-files'>
+            <title>Configuration Files</title>
+
+            <para>
+                Configuration files, which are denoted by the
+                <filename>.conf</filename> extension, define
+                various configuration variables that govern the project's build
+                process.
+                These files fall into several areas that define
+                machine configuration options, distribution configuration
+                options, compiler tuning options, general common
+                configuration options, and user configuration options.
+                The main configuration file is the sample
+                <filename>bitbake.conf</filename> file, which is
+                located within the BitBake source tree
+                <filename>conf</filename> directory.
+            </para>
+        </section>
+
+        <section id='classes'>
+            <title>Classes</title>
+
+            <para>
+                Class files, which are denoted by the
+                <filename>.bbclass</filename> extension, contain
+                information that is useful to share between metadata files.
+                The BitBake source tree currently comes with one class metadata file
+                called <filename>base.bbclass</filename>.
+                You can find this file in the
+                <filename>classes</filename> directory.
+                The <filename>base.bbclass</filename> class files is special since it
+                is always included automatically for all recipes
+                and classes.
+                This class contains definitions for standard basic tasks such
+                as fetching, unpacking, configuring (empty by default),
+                compiling (runs any Makefile present), installing (empty by
+                default) and packaging (empty by default).
+                These tasks are often overridden or extended by other classes
+                added during the project development process.
+            </para>
+        </section>
+
+        <section id='layers'>
+            <title>Layers</title>
+
+            <para>
+                Layers allow you to isolate different types of
+                customizations from each other.
+                While you might find it tempting to keep everything in one layer
+                when working on a single project, the more modular you organize
+                your metadata, the easier it is to cope with future changes.
+            </para>
+
+            <para>
+                To illustrate how you can use layers to keep things modular,
+                consider customizations you might make to support a specific target machine.
+                These types of customizations typically reside in a special layer,
+                rather than a general layer, called a Board Support Package (BSP)
+                Layer.
+                Furthermore, the machine customizations should be isolated from
+                recipes and metadata that support a new GUI environment, for
+                example.
+                This situation gives you a couple of layers: one for the machine
+                configurations and one for the GUI environment.
+                It is important to understand, however, that the BSP layer can still
+                make machine-specific additions to recipes within
+                the GUI environment layer without polluting the GUI layer itself
+                with those machine-specific changes.
+                You can accomplish this through a recipe that is a BitBake append
+                (<filename>.bbappend</filename>) file.
+            </para>
+        </section>
+
+        <section id='append-bbappend-files'>
+            <title>Append Files</title>
+
+            <para>
+                Append files, which are files that have the
+                <filename>.bbappend</filename> file extension, extend or
+                override information in an existing recipe file.
+            </para>
+
+            <para>
+                BitBake expects every append file to have a corresponding recipe file.
+                Furthermore, the append file and corresponding recipe file
+                must use the same root filename.
+                The filenames can differ only in the file type suffix used
+                (e.g. <filename>formfactor_0.0.bb</filename> and
+                <filename>formfactor_0.0.bbappend</filename>).
+            </para>
+
+            <para>
+                Information in append files extends or
+                overrides the information in the underlying,
+                similarly-named recipe files.
+            </para>
+
+            <para>
+                When you name an append file, you can use the
+                wildcard character (%) to allow for matching recipe names.
+                For example, suppose you have an append file named
+                as follows:
+                <literallayout class='monospaced'>
+     busybox_1.21.%.bbappend
+                </literallayout>
+                That append file would match any <filename>busybox_1.21.x.bb</filename>
+                version of the recipe.
+                So, the append file would match the following recipe names:
+                <literallayout class='monospaced'>
+     busybox_1.21.1.bb
+     busybox_1.21.2.bb
+     busybox_1.21.3.bb
+                </literallayout>
+                If the <filename>busybox</filename> recipe was updated to
+                <filename>busybox_1.3.0.bb</filename>, the append name would not
+                match.
+                However, if you named the append file
+                <filename>busybox_1.%.bbappend</filename>, then you would have a match.
+            </para>
+
+            <para>
+                In the most general case, you could name the append file something as
+                simple as <filename>busybox_%.bbappend</filename> to be entirely
+                version independent.
+            </para>
+        </section>
+    </section>
+
+    <section id='obtaining-bitbake'>
+        <title>Obtaining BitBake</title>
+
+        <para>
+            You can obtain BitBake several different ways:
+            <itemizedlist>
+                <listitem><para><emphasis>Cloning BitBake:</emphasis>
+                    Using Git to clone the BitBake source code repository
+                    is the recommended method for obtaining BitBake.
+                    Cloning the repository makes it easy to get bug fixes
+                    and have access to stable branches and the master
+                    branch.
+                    Once you have cloned BitBake, you should use
+                    the latest stable
+                    branch for development since the master branch is for
+                    BitBake development and might contain less stable changes.
+                    </para>
+                    <para>You usually need a version of BitBake
+                    that matches the metadata you are using.
+                    The metadata is generally backwards compatible but
+                    not forward compatible.</para>
+                    <para>Here is an example that clones the BitBake repository:
+                    <literallayout class='monospaced'>
+     $ git clone git://git.openembedded.org/bitbake
+                    </literallayout>
+                    This command clones the BitBake Git repository into a
+                    directory called <filename>bitbake</filename>.
+                    Alternatively, you can
+                    designate a directory after the
+                    <filename>git clone</filename> command
+                    if you want to call the new directory something
+                    other than <filename>bitbake</filename>.
+                    Here is an example that names the directory
+                    <filename>bbdev</filename>:
+                    <literallayout class='monospaced'>
+     $ git clone git://git.openembedded.org/bitbake bbdev
+                    </literallayout></para></listitem>
+                <listitem><para><emphasis>Installation using your Distribution
+                    Package Management System:</emphasis>
+                    This method is not
+                    recommended because the BitBake version that is
+                    provided by your distribution, in most cases,
+                    is several
+                    releases behind a snapshot of the BitBake repository.
+                    </para></listitem>
+                <listitem><para><emphasis>Taking a snapshot of BitBake:</emphasis>
+                    Downloading a snapshot of BitBake from the
+                    source code repository gives you access to a known
+                    branch or release of BitBake.
+                    <note>
+                         Cloning the Git repository, as described earlier,
+                         is the preferred method for getting BitBake.
+                         Cloning the repository makes it easier to update as
+                         patches are added to the stable branches.
+                    </note></para>
+                    <para>The following example downloads a snapshot of
+                    BitBake version 1.17.0:
+                    <literallayout class='monospaced'>
+     $ wget http://git.openembedded.org/bitbake/snapshot/bitbake-1.17.0.tar.gz
+     $ tar zxpvf bitbake-1.17.0.tar.gz
+                    </literallayout>
+                    After extraction of the tarball using the tar utility,
+                    you have a directory entitled
+                    <filename>bitbake-1.17.0</filename>.
+                    </para></listitem>
+                <listitem><para><emphasis>Using the BitBake that Comes With Your
+                    Build Checkout:</emphasis>
+                    A final possibility for getting a copy of BitBake is that it
+                    already comes with your checkout of a larger Bitbake-based build
+                    system, such as Poky.
+                    Rather than manually checking out individual layers and
+                    gluing them together yourself, you can check
+                    out an entire build system.
+                    The checkout will already include a version of BitBake that
+                    has been thoroughly tested for compatibility with the other
+                    components.
+                    For information on how to check out a particular BitBake-based
+                    build system, consult that build system's supporting documentation.
+                    </para></listitem>
+            </itemizedlist>
+        </para>
+    </section>
+
+    <section id="bitbake-user-manual-command">
+        <title>The BitBake Command</title>
+
+        <para>
+            The <filename>bitbake</filename> command is the primary interface
+            to the BitBake tool.
+            This section presents the BitBake command syntax and provides
+            several execution examples.
+        </para>
+
+        <section id='usage-and-syntax'>
+            <title>Usage and syntax</title>
+
+            <para>
+                Following is the usage and syntax for BitBake:
+                <literallayout class='monospaced'>
+     $ bitbake -h
+     Usage: bitbake [options] [recipename/target recipe:do_task ...]
+
+         Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
+         It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
+         will provide the layer, BBFILES and other configuration information.
+
+     Options:
+       --version             show program's version number and exit
+       -h, --help            show this help message and exit
+       -b BUILDFILE, --buildfile=BUILDFILE
+                             Execute tasks from a specific .bb recipe directly.
+                             WARNING: Does not handle any dependencies from other
+                             recipes.
+       -k, --continue        Continue as much as possible after an error. While the
+                             target that failed and anything depending on it cannot
+                             be built, as much as possible will be built before
+                             stopping.
+       -a, --tryaltconfigs   Continue with builds by trying to use alternative
+                             providers where possible.
+       -f, --force           Force the specified targets/task to run (invalidating
+                             any existing stamp file).
+       -c CMD, --cmd=CMD     Specify the task to execute. The exact options
+                             available depend on the metadata. Some examples might
+                             be 'compile' or 'populate_sysroot' or 'listtasks' may
+                             give a list of the tasks available.
+       -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP
+                             Invalidate the stamp for the specified task such as
+                             'compile' and then run the default task for the
+                             specified target(s).
+       -r PREFILE, --read=PREFILE
+                             Read the specified file before bitbake.conf.
+       -R POSTFILE, --postread=POSTFILE
+                             Read the specified file after bitbake.conf.
+       -v, --verbose         Enable tracing of shell tasks (with 'set -x').
+                             Also print bb.note(...) messages to stdout (in
+                             addition to writing them to ${T}/log.do_&lt;task&gt;).
+       -D, --debug           Increase the debug level. You can specify this
+                             more than once. -D sets the debug level to 1,
+                             where only bb.debug(1, ...) messages are printed
+                             to stdout; -DD sets the debug level to 2, where
+                             both bb.debug(1, ...) and bb.debug(2, ...)
+                             messages are printed; etc. Without -D, no debug
+                             messages are printed. Note that -D only affects
+                             output to stdout. All debug messages are written
+                             to ${T}/log.do_taskname, regardless of the debug
+                             level.
+       -n, --dry-run         Don't execute, just go through the motions.
+       -S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER
+                             Dump out the signature construction information, with
+                             no task execution. The SIGNATURE_HANDLER parameter is
+                             passed to the handler. Two common values are none and
+                             printdiff but the handler may define more/less. none
+                             means only dump the signature, printdiff means compare
+                             the dumped signature with the cached one.
+       -p, --parse-only      Quit after parsing the BB recipes.
+       -s, --show-versions   Show current and preferred versions of all recipes.
+       -e, --environment     Show the global or per-recipe environment complete
+                             with information about where variables were
+                             set/changed.
+       -g, --graphviz        Save dependency tree information for the specified
+                             targets in the dot syntax.
+       -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
+                             Assume these dependencies don't exist and are already
+                             provided (equivalent to ASSUME_PROVIDED). Useful to
+                             make dependency graphs more appealing
+       -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
+                             Show debug logging for the specified logging domains
+       -P, --profile         Profile the command and save reports.
+       -u UI, --ui=UI        The user interface to use (taskexp, knotty or
+                             ncurses - default knotty).
+       -t SERVERTYPE, --servertype=SERVERTYPE
+                             Choose which server type to use (process or xmlrpc -
+                             default process).
+       --token=XMLRPCTOKEN   Specify the connection token to be used when
+                             connecting to a remote server.
+       --revisions-changed   Set the exit code depending on whether upstream
+                             floating revisions have changed or not.
+       --server-only         Run bitbake without a UI, only starting a server
+                             (cooker) process.
+       -B BIND, --bind=BIND  The name/address for the bitbake server to bind to.
+       --no-setscene         Do not run any setscene tasks. sstate will be ignored
+                             and everything needed, built.
+       --setscene-only       Only run setscene tasks, don't run any real tasks.
+       --remote-server=REMOTE_SERVER
+                             Connect to the specified server.
+       -m, --kill-server     Terminate the remote server.
+       --observe-only        Connect to a server as an observing-only client.
+       --status-only         Check the status of the remote bitbake server.
+       -w WRITEEVENTLOG, --write-log=WRITEEVENTLOG
+                             Writes the event log of the build to a bitbake event
+                             json file. Use '' (empty string) to assign the name
+                             automatically.
+                </literallayout>
+            </para>
+        </section>
+
+        <section id='bitbake-examples'>
+            <title>Examples</title>
+
+            <para>
+                This section presents some examples showing how to use BitBake.
+            </para>
+
+            <section id='example-executing-a-task-against-a-single-recipe'>
+                <title>Executing a Task Against a Single Recipe</title>
+
+                <para>
+                    Executing tasks for a single recipe file is relatively simple.
+                    You specify the file in question, and BitBake parses
+                    it and executes the specified task.
+                    If you do not specify a task, BitBake executes the default
+                    task, which is "build”.
+                    BitBake obeys inter-task dependencies when doing
+                    so.
+                </para>
+
+                <para>
+                    The following command runs the build task, which is
+                    the default task, on the <filename>foo_1.0.bb</filename>
+                    recipe file:
+                    <literallayout class='monospaced'>
+     $ bitbake -b foo_1.0.bb
+                    </literallayout>
+                    The following command runs the clean task on the
+                    <filename>foo.bb</filename> recipe file:
+                    <literallayout class='monospaced'>
+     $ bitbake -b foo.bb -c clean
+                    </literallayout>
+                    <note>
+                        The "-b" option explicitly does not handle recipe
+                        dependencies.
+                        Other than for debugging purposes, it is instead
+                        recommended that you use the syntax presented in the
+                        next section.
+                    </note>
+                </para>
+            </section>
+
+            <section id='executing-tasks-against-a-set-of-recipe-files'>
+                <title>Executing Tasks Against a Set of Recipe Files</title>
+
+                <para>
+                    There are a number of additional complexities introduced
+                    when one wants to manage multiple <filename>.bb</filename>
+                    files.
+                    Clearly there needs to be a way to tell BitBake what
+                    files are available and, of those, which you
+                    want to execute.
+                    There also needs to be a way for each recipe
+                    to express its dependencies, both for build-time and
+                    runtime.
+                    There must be a way for you to express recipe preferences
+                    when multiple recipes provide the same functionality, or when
+                    there are multiple versions of a recipe.
+                </para>
+
+                <para>
+                    The <filename>bitbake</filename> command, when not using
+                    "--buildfile" or "-b" only accepts a "PROVIDES".
+                    You cannot provide anything else.
+                    By default, a recipe file generally "PROVIDES" its
+                    "packagename" as shown in the following example:
+                    <literallayout class='monospaced'>
+     $ bitbake foo
+                    </literallayout>
+                    This next example "PROVIDES" the package name and also uses
+                    the "-c" option to tell BitBake to just execute the
+                    <filename>do_clean</filename> task:
+                    <literallayout class='monospaced'>
+     $ bitbake -c clean foo
+                    </literallayout>
+                </para>
+            </section>
+
+            <section id='executing-a-list-of-task-and-recipe-combinations'>
+                <title>Executing a List of Task and Recipe Combinations</title>
+
+                <para>
+                    The BitBake command line supports specifying different
+                    tasks for individual targets when you specify multiple
+                    targets.
+                    For example, suppose you had two targets (or recipes)
+                    <filename>myfirstrecipe</filename> and
+                    <filename>mysecondrecipe</filename> and you needed
+                    BitBake to run <filename>taskA</filename> for the first
+                    recipe and <filename>taskB</filename> for the second
+                    recipe:
+                    <literallayout class='monospaced'>
+     $ bitbake myfirstrecipe:do_taskA mysecondrecipe:do_taskB
+                    </literallayout>
+                </para>
+            </section>
+
+            <section id='generating-dependency-graphs'>
+                <title>Generating Dependency Graphs</title>
+
+                <para>
+                    BitBake is able to generate dependency graphs using
+                    the <filename>dot</filename> syntax.
+                    You can convert these graphs into images using the
+                    <filename>dot</filename> tool from
+                    <ulink url='http://www.graphviz.org'>Graphviz</ulink>.
+                </para>
+
+                <para>
+                    When you generate a dependency graph, BitBake writes three files
+                    to the current working directory:
+                    <itemizedlist>
+                        <listitem><para>
+                            <emphasis><filename>recipe-depends.dot</filename>:</emphasis>
+                            Shows dependencies between recipes (i.e. a collapsed version of
+                            <filename>task-depends.dot</filename>).
+                            </para></listitem>
+                        <listitem><para>
+                            <emphasis><filename>task-depends.dot</filename>:</emphasis>
+                            Shows dependencies between tasks.
+                            These dependencies match BitBake's internal task execution list.
+                            </para></listitem>
+                        <listitem><para>
+                            <emphasis><filename>pn-buildlist</filename>:</emphasis>
+                            Shows a simple list of targets that are to be built.
+                            </para></listitem>
+                    </itemizedlist>
+                </para>
+
+                <para>
+                    To stop depending on common depends, use the "-I" depend
+                    option and BitBake omits them from the graph.
+                    Leaving this information out can produce more readable graphs.
+                    This way, you can remove from the graph
+                    <filename>DEPENDS</filename> from inherited classes
+                    such as <filename>base.bbclass</filename>.
+                </para>
+
+                <para>
+                    Here are two examples that create dependency graphs.
+                    The second example omits depends common in OpenEmbedded from
+                    the graph:
+                    <literallayout class='monospaced'>
+     $ bitbake -g foo
+
+     $ bitbake -g -I virtual/kernel -I eglibc foo
+                    </literallayout>
+                </para>
+            </section>
+        </section>
+    </section>
+</chapter>

bitbake/doc/bitbake-user-manual/bitbake-user-manual-style.css

@@ -0,0 +1,984 @@
+/*
+   Generic XHTML / DocBook XHTML CSS Stylesheet.
+
+   Browser wrangling and typographic design by
+      Oyvind Kolas / pippin@gimp.org
+
+   Customised for Poky by
+      Matthew Allum / mallum@o-hand.com
+
+   Thanks to:
+     Liam R. E. Quin
+     William Skaggs
+     Jakub Steiner
+
+   Structure
+   ---------
+
+   The stylesheet is divided into the following sections:
+
+       Positioning
+          Margins, paddings, width, font-size, clearing.
+       Decorations
+          Borders, style
+       Colors
+          Colors
+       Graphics
+          Graphical backgrounds
+       Nasty IE tweaks
+          Workarounds needed to make it work in internet explorer,
+          currently makes the stylesheet non validating, but up until
+          this point it is validating.
+       Mozilla extensions
+          Transparency for footer
+	  Rounded corners on boxes
+
+*/
+
+
+  /*************** /
+ /  Positioning   /
+/ ***************/
+
+body {
+  font-family: Verdana, Sans, sans-serif;
+
+  min-width: 640px;
+  width: 80%;
+  margin:  0em auto;
+  padding: 2em 5em 5em 5em;
+  color: #333;
+}
+
+h1,h2,h3,h4,h5,h6,h7 {
+  font-family: Arial, Sans;
+  color: #00557D;
+  clear: both;
+}
+
+h1 {
+  font-size: 2em;
+  text-align: left;
+  padding: 0em 0em 0em 0em;
+  margin: 2em 0em 0em 0em;
+}
+
+h2.subtitle {
+  margin: 0.10em 0em 3.0em 0em;
+  padding: 0em 0em 0em 0em;
+  font-size: 1.8em;
+  padding-left: 20%;
+  font-weight: normal;
+  font-style: italic;
+}
+
+h2 {
+  margin: 2em 0em 0.66em 0em;
+  padding: 0.5em 0em 0em 0em;
+  font-size: 1.5em;
+  font-weight: bold;
+}
+
+h3.subtitle {
+  margin: 0em 0em 1em 0em;
+  padding: 0em 0em 0em 0em;
+  font-size: 142.14%;
+  text-align: right;
+}
+
+h3 {
+  margin: 1em 0em 0.5em 0em;
+  padding: 1em 0em 0em 0em;
+  font-size: 140%;
+  font-weight: bold;
+}
+
+h4 {
+  margin: 1em 0em 0.5em 0em;
+  padding: 1em 0em 0em 0em;
+  font-size: 120%;
+  font-weight: bold;
+}
+
+h5 {
+  margin: 1em 0em 0.5em 0em;
+  padding: 1em 0em 0em 0em;
+  font-size: 110%;
+  font-weight: bold;
+}
+
+h6 {
+  margin: 1em 0em 0em 0em;
+  padding: 1em 0em 0em 0em;
+  font-size: 110%;
+  font-weight: bold;
+}
+
+.authorgroup {
+  background-color: transparent;
+  background-repeat: no-repeat;
+  padding-top: 256px;
+  background-image: url("figures/bitbake-title.png");
+  background-position: left top;
+  margin-top: -256px;
+  padding-right: 50px;
+  margin-left: 0px;
+  text-align: right;
+  width: 740px;
+}
+
+h3.author {
+  margin: 0em 0me 0em 0em;
+  padding: 0em 0em 0em 0em;
+  font-weight: normal;
+  font-size: 100%;
+  color: #333;
+  clear: both;
+}
+
+.author tt.email {
+  font-size: 66%;
+}
+
+.titlepage hr {
+  width: 0em;
+  clear: both;
+}
+
+.revhistory {
+  padding-top: 2em;
+  clear: both;
+}
+
+.toc,
+.list-of-tables,
+.list-of-examples,
+.list-of-figures {
+  padding: 1.33em 0em 2.5em 0em;
+  color: #00557D;
+}
+
+.toc p,
+.list-of-tables p,
+.list-of-figures p,
+.list-of-examples p {
+  padding: 0em 0em 0em 0em;
+  padding: 0em 0em 0.3em;
+  margin: 1.5em 0em 0em 0em;
+}
+
+.toc p b,
+.list-of-tables p b,
+.list-of-figures p b,
+.list-of-examples p b{
+  font-size: 100.0%;
+  font-weight: bold;
+}
+
+.toc dl,
+.list-of-tables dl,
+.list-of-figures dl,
+.list-of-examples dl {
+  margin: 0em 0em 0.5em 0em;
+  padding: 0em 0em 0em 0em;
+}
+
+.toc dt {
+  margin: 0em 0em 0em 0em;
+  padding: 0em 0em 0em 0em;
+}
+
+.toc dd {
+  margin: 0em 0em 0em 2.6em;
+  padding: 0em 0em 0em 0em;
+}
+
+div.glossary dl,
+div.variablelist dl {
+}
+
+.glossary dl dt,
+.variablelist dl dt,
+.variablelist dl dt span.term {
+  font-weight: normal;
+  width: 20em;
+  text-align: right;
+}
+
+.variablelist dl dt {
+  margin-top: 0.5em;
+}
+
+.glossary dl dd,
+.variablelist dl dd {
+  margin-top: -1em;
+  margin-left: 25.5em;
+}
+
+.glossary dd p,
+.variablelist dd p {
+  margin-top: 0em;
+  margin-bottom: 1em;
+}
+
+
+div.calloutlist table td {
+  padding: 0em 0em 0em 0em;
+  margin: 0em 0em 0em 0em;
+}
+
+div.calloutlist table td p {
+  margin-top: 0em;
+  margin-bottom: 1em;
+}
+
+div p.copyright {
+  text-align: left;
+}
+
+div.legalnotice p.legalnotice-title {
+  margin-bottom: 0em;
+}
+
+p {
+  line-height: 1.5em;
+  margin-top: 0em;
+
+}
+
+dl {
+  padding-top: 0em;
+}
+
+hr {
+  border: solid 1px;
+}
+
+
+.mediaobject,
+.mediaobjectco {
+  text-align: center;
+}
+
+img {
+  border: none;
+}
+
+ul {
+  padding: 0em 0em 0em 1.5em;
+}
+
+ul li {
+  padding: 0em 0em 0em 0em;
+}
+
+ul li p {
+  text-align: left;
+}
+
+table {
+  width :100%;
+}
+
+th {
+  padding: 0.25em;
+  text-align: left;
+  font-weight: normal;
+  vertical-align: top;
+}
+
+td {
+  padding: 0.25em;
+  vertical-align: top;
+}
+
+p a[id] {
+  margin: 0px;
+  padding: 0px;
+  display: inline;
+  background-image: none;
+}
+
+a {
+  text-decoration: underline;
+  color: #444;
+}
+
+pre {
+    overflow: auto;
+}
+
+a:hover {
+  text-decoration: underline;
+  /*font-weight: bold;*/
+}
+
+/* This style defines how the permalink character
+   appears by itself and when hovered over with
+   the mouse. */
+
+[alt='Permalink'] { color: #eee; }
+[alt='Permalink']:hover { color: black; }
+
+
+div.informalfigure,
+div.informalexample,
+div.informaltable,
+div.figure,
+div.table,
+div.example {
+  margin: 1em 0em;
+  padding: 1em;
+  page-break-inside: avoid;
+}
+
+
+div.informalfigure p.title b,
+div.informalexample p.title b,
+div.informaltable p.title b,
+div.figure p.title b,
+div.example p.title b,
+div.table p.title b{
+    padding-top: 0em;
+    margin-top: 0em;
+    font-size: 100%;
+    font-weight: normal;
+}
+
+.mediaobject .caption,
+.mediaobject .caption p  {
+  text-align: center;
+  font-size: 80%;
+  padding-top: 0.5em;
+  padding-bottom: 0.5em;
+}
+
+.epigraph {
+  padding-left: 55%;
+  margin-bottom: 1em;
+}
+
+.epigraph p {
+  text-align: left;
+}
+
+.epigraph .quote {
+  font-style: italic;
+}
+.epigraph .attribution {
+  font-style: normal;
+  text-align: right;
+}
+
+span.application {
+  font-style: italic;
+}
+
+.programlisting {
+  font-family: monospace;
+  font-size: 80%;
+  white-space: pre;
+  margin: 1.33em 0em;
+  padding: 1.33em;
+}
+
+.tip,
+.warning,
+.caution,
+.note {
+  margin-top: 1em;
+  margin-bottom: 1em;
+
+}
+
+/* force full width of table within div */
+.tip table,
+.warning table,
+.caution table,
+.note table {
+  border: none;
+  width: 100%;
+}
+
+
+.tip table th,
+.warning table th,
+.caution table th,
+.note table th {
+  padding: 0.8em 0.0em 0.0em 0.0em;
+  margin : 0em 0em 0em 0em;
+}
+
+.tip p,
+.warning p,
+.caution p,
+.note p {
+  margin-top: 0.5em;
+  margin-bottom: 0.5em;
+  padding-right: 1em;
+  text-align: left;
+}
+
+.acronym {
+  text-transform: uppercase;
+}
+
+b.keycap,
+.keycap {
+  padding: 0.09em 0.3em;
+  margin: 0em;
+}
+
+.itemizedlist li {
+  clear: none;
+}
+
+.filename {
+  font-size: medium;
+  font-family: Courier, monospace;
+}
+
+
+div.navheader, div.heading{
+  position: absolute;
+  left: 0em;
+  top: 0em;
+  width: 100%;
+  background-color: #cdf;
+  width: 100%;
+}
+
+div.navfooter, div.footing{
+  position: fixed;
+  left: 0em;
+  bottom: 0em;
+  background-color: #eee;
+  width: 100%;
+}
+
+
+div.navheader td,
+div.navfooter td {
+  font-size: 66%;
+}
+
+div.navheader table th {
+  /*font-family: Georgia, Times, serif;*/
+  /*font-size: x-large;*/
+  font-size: 80%;
+}
+
+div.navheader table {
+  border-left: 0em;
+  border-right: 0em;
+  border-top: 0em;
+  width: 100%;
+}
+
+div.navfooter table {
+  border-left: 0em;
+  border-right: 0em;
+  border-bottom: 0em;
+  width: 100%;
+}
+
+div.navheader table td a,
+div.navfooter table td a {
+  color: #777;
+  text-decoration: none;
+}
+
+/* normal text in the footer */
+div.navfooter table td {
+  color: black;
+}
+
+div.navheader table td a:visited,
+div.navfooter table td a:visited {
+  color: #444;
+}
+
+
+/* links in header and footer */
+div.navheader table td a:hover,
+div.navfooter table td a:hover {
+  text-decoration: underline;
+  background-color: transparent;
+  color: #33a;
+}
+
+div.navheader hr,
+div.navfooter hr {
+  display: none;
+}
+
+
+.qandaset tr.question td p {
+  margin: 0em 0em 1em 0em;
+  padding: 0em 0em 0em 0em;
+}
+
+.qandaset tr.answer td p {
+  margin: 0em 0em 1em 0em;
+  padding: 0em 0em 0em 0em;
+}
+.answer td {
+  padding-bottom: 1.5em;
+}
+
+.emphasis {
+  font-weight: bold;
+}
+
+
+  /************* /
+ / decorations  /
+/ *************/
+
+.titlepage {
+}
+
+.part .title {
+}
+
+.subtitle {
+    border: none;
+}
+
+/*
+h1 {
+  border: none;
+}
+
+h2 {
+  border-top: solid 0.2em;
+  border-bottom: solid 0.06em;
+}
+
+h3 {
+  border-top: 0em;
+  border-bottom: solid 0.06em;
+}
+
+h4 {
+  border: 0em;
+  border-bottom: solid 0.06em;
+}
+
+h5 {
+  border: 0em;
+}
+*/
+
+.programlisting {
+  border: solid 1px;
+}
+
+div.figure,
+div.table,
+div.informalfigure,
+div.informaltable,
+div.informalexample,
+div.example {
+  border: 1px solid;
+}
+
+
+
+.tip,
+.warning,
+.caution,
+.note {
+  border: 1px solid;
+}
+
+.tip table th,
+.warning table th,
+.caution table th,
+.note table th {
+  border-bottom: 1px solid;
+}
+
+.question td {
+  border-top: 1px solid black;
+}
+
+.answer {
+}
+
+
+b.keycap,
+.keycap {
+  border: 1px solid;
+}
+
+
+div.navheader, div.heading{
+  border-bottom: 1px solid;
+}
+
+
+div.navfooter, div.footing{
+  border-top: 1px solid;
+}
+
+  /********* /
+ /  colors  /
+/ *********/
+
+body {
+  color: #333;
+  background: white;
+}
+
+a {
+  background: transparent;
+}
+
+a:hover {
+  background-color: #dedede;
+}
+
+
+h1,
+h2,
+h3,
+h4,
+h5,
+h6,
+h7,
+h8 {
+  background-color: transparent;
+}
+
+hr {
+  border-color: #aaa;
+}
+
+
+.tip, .warning, .caution, .note {
+  border-color: #fff;
+}
+
+
+.tip table th,
+.warning table th,
+.caution table th,
+.note table th {
+  border-bottom-color: #fff;
+}
+
+
+.warning {
+  background-color: #f0f0f2;
+}
+
+.caution {
+  background-color: #f0f0f2;
+}
+
+.tip {
+  background-color: #f0f0f2;
+}
+
+.note {
+  background-color: #f0f0f2;
+}
+
+.glossary dl dt,
+.variablelist dl dt,
+.variablelist dl dt span.term {
+  color: #044;
+}
+
+div.figure,
+div.table,
+div.example,
+div.informalfigure,
+div.informaltable,
+div.informalexample {
+  border-color: #aaa;
+}
+
+pre.programlisting {
+  color: black;
+  background-color: #fff;
+  border-color: #aaa;
+  border-width: 2px;
+}
+
+.guimenu,
+.guilabel,
+.guimenuitem {
+  background-color: #eee;
+}
+
+
+b.keycap,
+.keycap {
+  background-color: #eee;
+  border-color: #999;
+}
+
+
+div.navheader {
+  border-color: black;
+}
+
+
+div.navfooter {
+  border-color: black;
+}
+
+
+  /*********** /
+ /  graphics  /
+/ ***********/
+
+/*
+body {
+  background-image: url("images/body_bg.jpg");
+  background-attachment: fixed;
+}
+
+.navheader,
+.note,
+.tip {
+  background-image: url("images/note_bg.jpg");
+  background-attachment: fixed;
+}
+
+.warning,
+.caution {
+  background-image: url("images/warning_bg.jpg");
+  background-attachment: fixed;
+}
+
+.figure,
+.informalfigure,
+.example,
+.informalexample,
+.table,
+.informaltable {
+  background-image: url("images/figure_bg.jpg");
+  background-attachment: fixed;
+}
+
+*/
+h1,
+h2,
+h3,
+h4,
+h5,
+h6,
+h7{
+}
+
+/*
+Example of how to stick an image as part of the title.
+
+div.article .titlepage .title
+{
+  background-image: url("figures/white-on-black.png");
+  background-position: center;
+  background-repeat: repeat-x;
+}
+*/
+
+div.preface .titlepage .title,
+div.colophon .title,
+div.chapter .titlepage .title,
+div.article .titlepage .title
+{
+}
+
+div.section div.section .titlepage .title,
+div.sect2 .titlepage .title {
+    background: none;
+}
+
+
+h1.title {
+  background-color: transparent;
+  background-repeat: no-repeat;
+  height: 256px;
+  text-indent: -9000px;
+  overflow:hidden;
+}
+
+h2.subtitle {
+  background-color: transparent;
+  text-indent: -9000px;
+  overflow:hidden;
+  width: 0px;
+  display: none;
+}
+
+  /*************************************** /
+ /  pippin.gimp.org specific alterations  /
+/ ***************************************/
+
+/*
+div.heading, div.navheader {
+  color: #777;
+  font-size: 80%;
+  padding: 0;
+  margin: 0;
+  text-align: left;
+  position: absolute;
+  top: 0px;
+  left: 0px;
+  width: 100%;
+  height: 50px;
+  background: url('/gfx/heading_bg.png') transparent;
+  background-repeat: repeat-x;
+  background-attachment: fixed;
+  border: none;
+}
+
+div.heading a {
+  color: #444;
+}
+
+div.footing, div.navfooter {
+  border: none;
+  color: #ddd;
+  font-size: 80%;
+  text-align:right;
+
+  width: 100%;
+  padding-top: 10px;
+  position: absolute;
+  bottom: 0px;
+  left: 0px;
+
+  background: url('/gfx/footing_bg.png') transparent;
+}
+*/
+
+
+
+  /****************** /
+ /  nasty ie tweaks  /
+/ ******************/
+
+/*
+div.heading, div.navheader {
+  width:expression(document.body.clientWidth + "px");
+}
+
+div.footing, div.navfooter {
+  width:expression(document.body.clientWidth + "px");
+  margin-left:expression("-5em");
+}
+body {
+  padding:expression("4em 5em 0em 5em");
+}
+*/
+
+  /**************************************** /
+ / mozilla vendor specific css extensions  /
+/ ****************************************/
+/*
+div.navfooter, div.footing{
+  -moz-opacity: 0.8em;
+}
+
+div.figure,
+div.table,
+div.informalfigure,
+div.informaltable,
+div.informalexample,
+div.example,
+.tip,
+.warning,
+.caution,
+.note {
+  -moz-border-radius: 0.5em;
+}
+
+b.keycap,
+.keycap {
+  -moz-border-radius: 0.3em;
+}
+*/
+
+table tr td table tr td {
+  display: none;
+}
+
+
+hr {
+  display: none;
+}
+
+table {
+  border: 0em;
+}
+
+ .photo {
+  float: right;
+  margin-left:   1.5em;
+  margin-bottom: 1.5em;
+  margin-top: 0em;
+  max-width:      17em;
+  border:     1px solid gray;
+  padding:    3px;
+  background: white;
+}
+ .seperator {
+   padding-top: 2em;
+   clear: both;
+  }
+
+  #validators {
+      margin-top: 5em;
+      text-align: right;
+      color: #777;
+  }
+  @media print {
+      body {
+          font-size: 8pt;
+      }
+      .noprint {
+          display: none;
+      }
+  }
+
+
+.tip,
+.note {
+   background: #f0f0f2;
+   color: #333;
+   padding: 20px;
+   margin: 20px;
+}
+
+.tip h3,
+.note h3 {
+   padding: 0em;
+   margin: 0em;
+   font-size: 2em;
+   font-weight: bold;
+   color: #333;
+}
+
+.tip a,
+.note a {
+   color: #333;
+   text-decoration: underline;
+}
+
+.footnote {
+   font-size: small;
+   color: #333;
+}
+
+/* Changes the announcement text */
+.tip h3,
+.warning h3,
+.caution h3,
+.note h3 {
+   font-size:large;
+   color: #00557D;
+}

bitbake/doc/bitbake-user-manual/bitbake-user-manual.xml

@@ -0,0 +1,88 @@
+<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+
+<book id='bitbake-user-manual' lang='en'
+        xmlns:xi="http://www.w3.org/2003/XInclude"
+        xmlns="http://docbook.org/ns/docbook"
+        >
+    <bookinfo>
+
+        <mediaobject>
+            <imageobject>
+                <imagedata fileref='figures/bitbake-title.png'
+                    format='SVG'
+                    align='left' scalefit='1' width='100%'/>
+            </imageobject>
+        </mediaobject>
+
+        <title>
+            BitBake User Manual
+        </title>
+
+        <authorgroup>
+            <author>
+                <firstname>Richard Purdie, Chris Larson, and </firstname> <surname>Phil Blundell</surname>
+                <affiliation>
+                    <orgname>BitBake Community</orgname>
+                </affiliation>
+                <email>bitbake-devel@lists.openembedded.org</email>
+            </author>
+        </authorgroup>
+
+<!--
+# Add in some revision history if we want it here.
+        <revhistory>
+            <revision>
+                <revnumber>x.x</revnumber>
+                <date>dd month year</date>
+                <revremark>Some relevent comment</revremark>
+            </revision>
+            <revision>
+                <revnumber>x.x</revnumber>
+                <date>dd month year</date>
+                <revremark>Some relevent comment</revremark>
+            </revision>
+            <revision>
+                <revnumber>x.x</revnumber>
+                <date>dd month year</date>
+                <revremark>Some relevent comment</revremark>
+            </revision>
+            <revision>
+                <revnumber>x.x</revnumber>
+                <date>dd month year</date>
+                <revremark>Some relevent comment</revremark>
+            </revision>
+       </revhistory>
+-->
+
+        <copyright>
+            <year>2004-2016</year>
+            <holder>Richard Purdie</holder>
+            <holder>Chris Larson</holder>
+            <holder>and Phil Blundell</holder>
+        </copyright>
+
+        <legalnotice>
+            <para>
+                This work is licensed under the Creative Commons Attribution License.
+                To view a copy of this license, visit
+                <ulink url="http://creativecommons.org/licenses/by/2.5/">http://creativecommons.org/licenses/by/2.5/</ulink>
+                or send a letter to Creative Commons, 444 Castro Street,
+                Suite 900, Mountain View, California 94041, USA.
+            </para>
+        </legalnotice>
+    </bookinfo>
+
+    <xi:include href="bitbake-user-manual-intro.xml"/>
+
+    <xi:include href="bitbake-user-manual-execution.xml"/>
+
+    <xi:include href="bitbake-user-manual-metadata.xml"/>
+
+    <xi:include href="bitbake-user-manual-fetching.xml"/>
+
+    <xi:include href="bitbake-user-manual-ref-variables.xml"/>
+
+    <xi:include href="bitbake-user-manual-hello.xml"/>
+
+</book>

bitbake/doc/bitbake-user-manual/html.css

@@ -0,0 +1,281 @@
+/* Feuille de style DocBook du projet Traduc.org                */
+/* DocBook CSS stylesheet of the Traduc.org project             */
+
+/* (c) Jean-Philippe Gu<47>rard - 14 ao<61>t 2004                     */
+/* (c) Jean-Philippe Gu<47>rard - 14 August 2004                   */
+
+/* Cette feuille de style est libre, vous pouvez la             */
+/* redistribuer et la modifier selon les termes de la Licence   */
+/* Art Libre. Vous trouverez un exemplaire de cette Licence sur */
+/* http://tigreraye.org/Petit-guide-du-traducteur.html#licence-art-libre */
+
+/* This work of art is free, you can redistribute it and/or     */
+/* modify it according to terms of the Free Art license. You    */
+/* will find a specimen of this license on the Copyleft         */
+/* Attitude web site: http://artlibre.org as well as on other   */
+/* sites.                                                       */
+/* Please note that the French version of this licence as shown */
+/* on http://tigreraye.org/Petit-guide-du-traducteur.html#licence-art-libre */
+/* is only official licence of this document. The English       */
+/* is only provided to help you understand this licence.        */
+
+/* La derni<6E>re version de cette feuille de style est toujours   */
+/* disponible sur<75>: http://tigreraye.org/style.css              */
+/* Elle est <20>galement disponible sur<75>:                          */
+/* http://www.traduc.org/docs/HOWTO/lecture/style.css           */
+
+/* The latest version of this stylesheet is available from:     */
+/* http://tigreraye.org/style.css                               */
+/* It is also available on:                                     */
+/* http://www.traduc.org/docs/HOWTO/lecture/style.css           */
+
+/* N'h<>sitez pas <20> envoyer vos commentaires et corrections <20>    */
+/* Jean-Philippe Gu<47>rard <jean-philippe.guerard@tigreraye.org>  */
+
+/* Please send feedback and bug reports to                      */
+/* Jean-Philippe Gu<47>rard <jean-philippe.guerard@tigreraye.org>  */
+
+/* $Id: style.css,v 1.14 2004/09/10 20:12:09 fevrier Exp fevrier $ */
+
+/* Pr<50>sentation g<>n<EFBFBD>rale du document */
+/* Overall document presentation */
+
+body {
+    /*
+    font-family: Apolline, "URW Palladio L", Garamond, jGaramond,
+                 "Bitstream Cyberbit", "Palatino Linotype", serif;
+     */
+    margin: 7%;
+    background-color: white;
+}
+
+/* Taille du texte */
+/* Text size */
+
+* { font-size: 100%; }
+
+/* Gestion des textes mis en relief imbriqu<71>s */
+/* Embedded emphasis */
+
+em { font-style: italic; }
+em em { font-style: normal; }
+em em em { font-style: italic; }
+
+/* Titres */
+/* Titles */
+
+h1 { font-size: 200%; font-weight: 900; }
+h2 { font-size: 160%; font-weight: 900; }
+h3 { font-size: 130%; font-weight: bold; }
+h4 { font-size: 115%; font-weight: bold; }
+h5 { font-size: 108%; font-weight: bold; }
+h6 {                  font-weight: bold; }
+
+/* Nom de famille en petites majuscules (uniquement en fran<61>ais) */
+/* Last names in small caps (for French only) */
+
+*[class~="surname"]:lang(fr) { font-variant: small-caps; }
+
+/* Blocs de citation */
+/* Quotation blocs */
+
+div[class~="blockquote"] {
+  border: solid 2px #AAA;
+  padding: 5px;
+  margin: 5px;
+}
+
+div[class~="blockquote"] > table {
+  border: none;
+}
+
+/* Blocs lit<69>raux<75>: fond gris clair */
+/* Literal blocs: light gray background */
+
+*[class~="literallayout"] {
+  background: #f0f0f0;
+  padding: 5px;
+  margin: 5px;
+}
+
+/* Programmes et captures texte<74>: fond bleu clair */
+/* Listing and text screen snapshots: light blue background */
+
+*[class~="programlisting"], *[class~="screen"] {
+  background: #f0f0ff;
+  padding: 5px;
+  margin: 5px;
+}
+
+/* Les textes <20> remplacer sont surlign<67>s en vert p<>le */
+/* Replaceable text in highlighted in pale green */
+
+*[class~="replaceable"] { 
+    background-color: #98fb98;
+    font-style: normal; }
+
+/* Tables<65>: fonds gris clair & bords simples */
+/* Tables: light gray background and solid borders */
+
+*[class~="table"] *[class~="title"] { width:100%; border: 0px; }
+
+table {
+    border: 1px solid #aaa;
+    border-collapse: collapse;
+    padding: 2px;
+    margin: 5px;
+}
+
+/* Listes simples en style table */
+/* Simples lists in table presentation */
+
+table[class~="simplelist"] {
+    background-color: #F0F0F0;
+    margin: 5px;
+    border: solid 1px #AAA;
+}
+
+table[class~="simplelist"] td {
+    border: solid 1px #AAA;
+}
+
+/* Les tables */
+/* Tables */
+
+*[class~="table"] table {
+    background-color: #F0F0F0;
+    border: solid 1px #AAA;
+}
+*[class~="informaltable"] table { background-color: #F0F0F0; }
+
+th,td {
+    vertical-align: baseline;
+    text-align: left;
+    padding: 0.1em 0.3em;
+    empty-cells: show; 
+}
+
+/* Alignement des colonnes */
+/* Colunms alignment */
+
+td[align=center] ,  th[align=center]  { text-align: center; }
+td[align=right] ,   th[align=right]   { text-align: right; }
+td[align=left] ,    th[align=left]    { text-align: left; }
+td[align=justify] , th[align=justify] { text-align: justify; }
+
+/* Pas de marge autour des images */
+/* No inside margins for images */
+
+img { border: 0; }
+
+/* Les liens ne sont pas soulign<67>s */
+/* No underlines for links */
+
+:link , :visited , :active { text-decoration: none; }
+
+/* Prudence<63>: cadre jaune et fond jaune clair */
+/* Caution: yellow border and light yellow background */
+
+*[class~="caution"] {
+    border: solid 2px yellow;
+    background-color: #ffffe0;
+    padding: 1em 6px 1em ;
+    margin: 5px;
+}
+
+*[class~="caution"] th {
+    vertical-align: middle
+}
+
+*[class~="caution"] table {
+    background-color: #ffffe0;
+    border: none;
+}
+
+/* Note importante<74>: cadre jaune et fond jaune clair */
+/* Important: yellow border and light yellow background */
+
+*[class~="important"] {
+    border: solid 2px yellow;
+    background-color: #ffffe0;
+    padding: 1em 6px 1em;
+    margin: 5px;
+}
+
+*[class~="important"] th {
+    vertical-align: middle
+}
+
+*[class~="important"] table  {
+    background-color: #ffffe0;
+    border: none;
+}
+
+/* Mise en <20>vidence<63>: texte l<>g<EFBFBD>rement plus grand */
+/* Highlights: slightly larger texts */
+
+*[class~="highlights"] {
+    font-size:  110%;
+}
+
+/* Note<74>: cadre bleu et fond bleu clair */
+/* Notes: blue border and light blue background */
+
+*[class~="note"]   {
+    border: solid 2px #7099C5;
+    background-color: #f0f0ff;
+    padding: 1em 6px 1em ;
+    margin: 5px;
+}
+
+*[class~="note"] th {
+    vertical-align: middle
+}
+
+*[class~="note"] table {
+    background-color: #f0f0ff;
+    border: none;
+}
+
+/* Astuce<63>: cadre vert et fond vert clair */
+/* Tip: green border and light green background */
+
+*[class~="tip"] {
+    border: solid 2px #00ff00;
+    background-color: #f0ffff;
+    padding: 1em 6px 1em ;
+    margin: 5px;
+}
+
+*[class~="tip"] th {
+    vertical-align: middle;
+}
+
+*[class~="tip"] table {
+    background-color: #f0ffff;
+    border: none;
+}
+
+/* Avertissement<6E>: cadre rouge et fond rouge clair */
+/* Warning: red border and light red background */
+
+*[class~="warning"] {
+    border: solid 2px #ff0000;
+    background-color: #fff0f0; 
+    padding: 1em 6px 1em ;
+    margin: 5px;
+}
+
+*[class~="warning"] th {
+    vertical-align: middle;
+}
+                    
+
+*[class~="warning"] table {
+    background-color: #fff0f0;
+    border: none;
+}
+
+/* Fin */
+/* The End */
+

bitbake/doc/conf.py

@@ -1,101 +0,0 @@
-# Configuration file for the Sphinx documentation builder.
-#
-# This file only contains a selection of the most common options. For a full
-# list see the documentation:
-# https://www.sphinx-doc.org/en/master/usage/configuration.html
-
-# -- Path setup --------------------------------------------------------------
-
-# If extensions (or modules to document with autodoc) are in another directory,
-# add these directories to sys.path here. If the directory is relative to the
-# documentation root, use os.path.abspath to make it absolute, like shown here.
-#
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
-
-import sys
-import datetime
-
-current_version = "dev"
-
-# String used in sidebar
-version = 'Version: ' + current_version
-if current_version == 'dev':
-    version = 'Version: Current Development'
-# Version seen in documentation_options.js and hence in js switchers code
-release = current_version
-
-# -- Project information -----------------------------------------------------
-
-project = 'Bitbake'
-copyright = '2004-%s, Richard Purdie, Chris Larson, and Phil Blundell' \
-    % datetime.datetime.now().year
-author = 'Richard Purdie, Chris Larson, and Phil Blundell'
-
-# external links and substitutions
-extlinks = {
-    'yocto_docs': ('https://docs.yoctoproject.org%s', None),
-    'oe_lists': ('https://lists.openembedded.org%s', None),
-}
-
-# -- General configuration ---------------------------------------------------
-
-# Add any Sphinx extension module names here, as strings. They can be
-# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
-# ones.
-extensions = [
-    'sphinx.ext.autosectionlabel',
-    'sphinx.ext.extlinks',
-]
-autosectionlabel_prefix_document = True
-
-# Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
-
-# List of patterns, relative to source directory, that match files and
-# directories to ignore when looking for source files.
-# This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
-
-# master document name. The default changed from contents to index. so better
-# set it ourselves.
-master_doc = 'index'
-
-# create substitution for project configuration variables
-rst_prolog = """
-.. |project_name| replace:: %s
-.. |copyright| replace:: %s
-.. |author| replace:: %s
-""" % (project, copyright, author)
-
-# -- Options for HTML output -------------------------------------------------
-
-# The theme to use for HTML and HTML Help pages.  See the documentation for
-# a list of builtin themes.
-#
-try:
-    import sphinx_rtd_theme
-    html_theme = 'sphinx_rtd_theme'
-except ImportError:
-    sys.stderr.write("The Sphinx sphinx_rtd_theme HTML theme was not found.\
-    \nPlease make sure to install the sphinx_rtd_theme python package.\n")
-    sys.exit(1)
-
-# Add any paths that contain custom static files (such as style sheets) here,
-# relative to this directory. They are copied after the builtin static files,
-# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['sphinx-static']
-
-# Add customm CSS and JS files
-html_css_files = ['theme_overrides.css']
-html_js_files = ['switchers.js']
-
-# Hide 'Created using Sphinx' text
-html_show_sphinx = False
-
-# Add 'Last updated' on each page
-html_last_updated_fmt = '%b %d, %Y'
-
-# Remove the trailing 'dot' in section numbers
-html_secnumber_suffix = " "

bitbake/doc/genindex.rst

@@ -1,3 +0,0 @@
-=====
-Index
-=====

bitbake/doc/index.rst

@@ -1,38 +0,0 @@
-.. SPDX-License-Identifier: CC-BY-2.5
-
-===================
-BitBake User Manual
-===================
-
-|
-
-.. toctree::
-   :caption: Table of Contents
-   :numbered:
-
-   bitbake-user-manual/bitbake-user-manual-intro
-   bitbake-user-manual/bitbake-user-manual-execution
-   bitbake-user-manual/bitbake-user-manual-metadata
-   bitbake-user-manual/bitbake-user-manual-fetching
-   bitbake-user-manual/bitbake-user-manual-ref-variables
-   bitbake-user-manual/bitbake-user-manual-hello
-
-.. toctree::
-   :maxdepth: 1
-   :hidden:
-
-   genindex
-   releases
-
-----
-
-.. include:: <xhtml1-lat1.txt>
-
-| BitBake Community
-| Copyright |copy| |copyright|
-| <bitbake-devel@lists.openembedded.org>
-
-This work is licensed under the Creative Commons Attribution License.  To view a
-copy of this license, visit http://creativecommons.org/licenses/by/2.5/ or send
-a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain View,
-California 94041, USA.

bitbake/doc/poky.ent

@@ -0,0 +1,59 @@
+<!ENTITY DISTRO "1.4">
+<!ENTITY DISTRO_NAME "tbd">
+<!ENTITY YOCTO_DOC_VERSION "1.4">
+<!ENTITY POKYVERSION "8.0">
+<!ENTITY YOCTO_POKY "poky-&DISTRO_NAME;-&POKYVERSION;">
+<!ENTITY COPYRIGHT_YEAR "2010-2013">
+<!ENTITY YOCTO_DL_URL "http://downloads.yoctoproject.org">
+<!ENTITY YOCTO_HOME_URL "http://www.yoctoproject.org">
+<!ENTITY YOCTO_LISTS_URL "http://lists.yoctoproject.org">
+<!ENTITY YOCTO_BUGZILLA_URL "http://bugzilla.yoctoproject.org">
+<!ENTITY YOCTO_WIKI_URL "https://wiki.yoctoproject.org">
+<!ENTITY YOCTO_AB_URL "http://autobuilder.yoctoproject.org">
+<!ENTITY YOCTO_GIT_URL "http://git.yoctoproject.org">
+<!ENTITY YOCTO_ADTREPO_URL "http://adtrepo.yoctoproject.org">
+<!ENTITY OE_HOME_URL "http://www.openembedded.org">
+<!ENTITY OE_LISTS_URL "http://lists.linuxtogo.org/cgi-bin/mailman">
+<!ENTITY OE_DOCS_URL "http://docs.openembedded.org">
+<!ENTITY OH_HOME_URL "http://o-hand.com">
+<!ENTITY BITBAKE_HOME_URL "http://developer.berlios.de/projects/bitbake/">
+<!ENTITY ECLIPSE_MAIN_URL "http://www.eclipse.org/downloads">
+<!ENTITY ECLIPSE_DL_URL "http://download.eclipse.org">
+<!ENTITY ECLIPSE_DL_PLUGIN_URL "&YOCTO_DL_URL;/releases/eclipse-plugin/&DISTRO;">
+<!ENTITY ECLIPSE_UPDATES_URL "&ECLIPSE_DL_URL;/tm/updates/3.3">
+<!ENTITY ECLIPSE_INDIGO_URL "&ECLIPSE_DL_URL;/releases/indigo">
+<!ENTITY ECLIPSE_JUNO_URL "&ECLIPSE_DL_URL;/releases/juno">
+<!ENTITY ECLIPSE_INDIGO_CDT_URL "&ECLIPSE_DL_URL;tools/cdt/releases/indigo">
+<!ENTITY YOCTO_DOCS_URL "&YOCTO_HOME_URL;/docs">
+<!ENTITY YOCTO_SOURCES_URL "&YOCTO_HOME_URL;/sources/">
+<!ENTITY YOCTO_AB_PORT_URL "&YOCTO_AB_URL;:8010">
+<!ENTITY YOCTO_AB_NIGHTLY_URL "&YOCTO_AB_URL;/nightly/">
+<!ENTITY YOCTO_POKY_URL "&YOCTO_DL_URL;/releases/poky/">
+<!ENTITY YOCTO_RELEASE_DL_URL "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;">
+<!ENTITY YOCTO_TOOLCHAIN_DL_URL "&YOCTO_RELEASE_DL_URL;/toolchain/">
+<!ENTITY YOCTO_ECLIPSE_DL_URL "&YOCTO_RELEASE_DL_URL;/eclipse-plugin/indigo;">
+<!ENTITY YOCTO_ADTINSTALLER_DL_URL "&YOCTO_RELEASE_DL_URL;/adt_installer">
+<!ENTITY YOCTO_POKY_DL_URL "&YOCTO_RELEASE_DL_URL;/&YOCTO_POKY;.tar.bz2">
+<!ENTITY YOCTO_MACHINES_DL_URL "&YOCTO_RELEASE_DL_URL;/machines">
+<!ENTITY YOCTO_QEMU_DL_URL "&YOCTO_MACHINES_DL_URL;/qemu">
+<!ENTITY YOCTO_PYTHON-i686_DL_URL "&YOCTO_DL_URL;/releases/miscsupport/python-nativesdk-standalone-i686.tar.bz2">
+<!ENTITY YOCTO_PYTHON-x86_64_DL_URL "&YOCTO_DL_URL;/releases/miscsupport/python-nativesdk-standalone-x86_64.tar.bz2">
+<!ENTITY YOCTO_DOCS_QS_URL "&YOCTO_DOCS_URL;/&YOCTO_DOC_VERSION;/yocto-project-qs/yocto-project-qs.html">
+<!ENTITY YOCTO_DOCS_ADT_URL "&YOCTO_DOCS_URL;/&YOCTO_DOC_VERSION;/adt-manual/adt-manual.html">
+<!ENTITY YOCTO_DOCS_REF_URL "&YOCTO_DOCS_URL;/&YOCTO_DOC_VERSION;/ref-manual/ref-manual.html">
+<!ENTITY YOCTO_DOCS_BSP_URL "&YOCTO_DOCS_URL;/&YOCTO_DOC_VERSION;/bsp-guide/bsp-guide.html">
+<!ENTITY YOCTO_DOCS_DEV_URL "&YOCTO_DOCS_URL;/&YOCTO_DOC_VERSION;/dev-manual/dev-manual.html">
+<!ENTITY YOCTO_DOCS_KERNEL_URL "&YOCTO_DOCS_URL;/&YOCTO_DOC_VERSION;/kernel-manual/kernel-manual.html">
+<!ENTITY YOCTO_ADTPATH_DIR "/opt/poky/&DISTRO;">
+<!ENTITY YOCTO_POKY_TARBALL "&YOCTO_POKY;.tar.bz2">
+<!ENTITY OE_INIT_PATH "&YOCTO_POKY;/oe-init-build-env">
+<!ENTITY OE_INIT_FILE "oe-init-build-env">
+<!ENTITY UBUNTU_HOST_PACKAGES_ESSENTIAL "gawk wget git-core diffstat unzip texinfo \
+     build-essential chrpath">
+<!ENTITY FEDORA_HOST_PACKAGES_ESSENTIAL "gawk make wget tar bzip2 gzip python unzip perl patch \
+     diffutils diffstat git cpp gcc gcc-c++ eglibc-devel texinfo chrpath \
+     ccache">
+<!ENTITY OPENSUSE_HOST_PACKAGES_ESSENTIAL "python gcc gcc-c++ git chrpath make wget python-xml \
+     diffstat texinfo python-curses">
+<!ENTITY CENTOS_HOST_PACKAGES_ESSENTIAL "gawk make wget tar bzip2 gzip python unzip perl patch \
+     diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath">

bitbake/doc/releases.rst

@@ -1,130 +0,0 @@
-.. SPDX-License-Identifier: CC-BY-2.5
-
-=========================
- Current Release Manuals
-=========================
-
-****************************
-3.1 'dunfell' Release Series
-****************************
-
-- :yocto_docs:`3.1 BitBake User Manual </3.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`3.1.1 BitBake User Manual </3.1.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`3.1.2 BitBake User Manual </3.1.2/bitbake-user-manual/bitbake-user-manual.html>`
-
-==========================
- Previous Release Manuals
-==========================
-
-*************************
-3.0 'zeus' Release Series
-*************************
-
-- :yocto_docs:`3.0 BitBake User Manual </3.0/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`3.0.1 BitBake User Manual </3.0.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`3.0.2 BitBake User Manual </3.0.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`3.0.3 BitBake User Manual </3.0.3/bitbake-user-manual/bitbake-user-manual.html>`
-
-****************************
-2.7 'warrior' Release Series
-****************************
-
-- :yocto_docs:`2.7 BitBake User Manual </2.7/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.7.1 BitBake User Manual </2.7.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.7.2 BitBake User Manual </2.7.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.7.3 BitBake User Manual </2.7.3/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.7.4 BitBake User Manual </2.7.4/bitbake-user-manual/bitbake-user-manual.html>`
-
-*************************
-2.6 'thud' Release Series
-*************************
-
-- :yocto_docs:`2.6 BitBake User Manual </2.6/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.6.1 BitBake User Manual </2.6.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.6.2 BitBake User Manual </2.6.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.6.3 BitBake User Manual </2.6.3/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.6.4 BitBake User Manual </2.6.4/bitbake-user-manual/bitbake-user-manual.html>`
-
-*************************
-2.5 'sumo' Release Series
-*************************
-
-- :yocto_docs:`2.5 BitBake User Manual </2.5/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.5.1 BitBake User Manual </2.5.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.5.2 BitBake User Manual </2.5.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.5.3 BitBake User Manual </2.5.3/bitbake-user-manual/bitbake-user-manual.html>`
-
-**************************
-2.4 'rocko' Release Series
-**************************
-
-- :yocto_docs:`2.4 BitBake User Manual </2.4/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.4.1 BitBake User Manual </2.4.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.4.2 BitBake User Manual </2.4.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.4.3 BitBake User Manual </2.4.3/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.4.4 BitBake User Manual </2.4.4/bitbake-user-manual/bitbake-user-manual.html>`
-
-*************************
-2.3 'pyro' Release Series
-*************************
-
-- :yocto_docs:`2.3 BitBake User Manual </2.3/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.3.1 BitBake User Manual </2.3.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.3.2 BitBake User Manual </2.3.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.3.3 BitBake User Manual </2.3.3/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.3.4 BitBake User Manual </2.3.4/bitbake-user-manual/bitbake-user-manual.html>`
-
-**************************
-2.2 'morty' Release Series
-**************************
-
-- :yocto_docs:`2.2 BitBake User Manual </2.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.2.1 BitBake User Manual </2.2.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.2.2 BitBake User Manual </2.2.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.2.3 BitBake User Manual </2.2.3/bitbake-user-manual/bitbake-user-manual.html>`
-
-****************************
-2.1 'krogoth' Release Series
-****************************
-
-- :yocto_docs:`2.1 BitBake User Manual </2.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.1.1 BitBake User Manual </2.1.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.1.2 BitBake User Manual </2.1.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.1.3 BitBake User Manual </2.1.3/bitbake-user-manual/bitbake-user-manual.html>`
-
-***************************
-2.0 'jethro' Release Series
-***************************
-
-- :yocto_docs:`1.9 BitBake User Manual </1.9/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.0 BitBake User Manual </2.0/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.0.1 BitBake User Manual </2.0.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.0.2 BitBake User Manual </2.0.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`2.0.3 BitBake User Manual </2.0.3/bitbake-user-manual/bitbake-user-manual.html>`
-
-*************************
-1.8 'fido' Release Series
-*************************
-
-- :yocto_docs:`1.8 BitBake User Manual </1.8/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.8.1 BitBake User Manual </1.8.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.8.2 BitBake User Manual </1.8.2/bitbake-user-manual/bitbake-user-manual.html>`
-
-**************************
-1.7 'dizzy' Release Series
-**************************
-
-- :yocto_docs:`1.7 BitBake User Manual </1.7/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.7.1 BitBake User Manual </1.7.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.7.2 BitBake User Manual </1.7.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.7.3 BitBake User Manual </1.7.3/bitbake-user-manual/bitbake-user-manual.html>`
-
-**************************
-1.6 'daisy' Release Series
-**************************
-
-- :yocto_docs:`1.6 BitBake User Manual </1.6/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.6.1 BitBake User Manual </1.6.1/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.6.2 BitBake User Manual </1.6.2/bitbake-user-manual/bitbake-user-manual.html>`
-- :yocto_docs:`1.6.3 BitBake User Manual </1.6.3/bitbake-user-manual/bitbake-user-manual.html>`
-

bitbake/doc/sphinx-static/switchers.js

@@ -1,233 +0,0 @@
-(function() {
-  'use strict';
-
-  var all_versions = {
-    'dev': 'dev (3.2)',
-    '3.1.2': '3.1.2',
-    '3.0.3': '3.0.3',
-    '2.7.4': '2.7.4',
-  };
-
-  var all_doctypes = {
-      'single': 'Individual Webpages',
-      'mega': "All-in-one 'Mega' Manual",
-  };
-
-  // Simple version comparision
-  // Return 1 if a > b
-  // Return -1 if a < b
-  // Return 0 if a == b
-  function ver_compare(a, b) {
-    if (a == "dev") {
-       return 1;
-    }
-
-    if (a === b) {
-       return 0;
-    }
-
-    var a_components = a.split(".");
-    var b_components = b.split(".");
-
-    var len = Math.min(a_components.length, b_components.length);
-
-    // loop while the components are equal
-    for (var i = 0; i < len; i++) {
-        // A bigger than B
-        if (parseInt(a_components[i]) > parseInt(b_components[i])) {
-            return 1;
-        }
-
-        // B bigger than A
-        if (parseInt(a_components[i]) < parseInt(b_components[i])) {
-            return -1;
-        }
-    }
-
-    // If one's a prefix of the other, the longer one is greater.
-    if (a_components.length > b_components.length) {
-        return 1;
-    }
-
-    if (a_components.length < b_components.length) {
-        return -1;
-    }
-
-    // Otherwise they are the same.
-    return 0;
-  }
-
-  function build_version_select(current_series, current_version) {
-    var buf = ['<select>'];
-
-    $.each(all_versions, function(version, title) {
-      var series = version.substr(0, 3);
-      if (series == current_series) {
-        if (version == current_version)
-            buf.push('<option value="' + version + '" selected="selected">' + title + '</option>');
-        else
-            buf.push('<option value="' + version + '">' + title + '</option>');
-
-        if (version != current_version)
-            buf.push('<option value="' + current_version + '" selected="selected">' + current_version + '</option>');
-      } else {
-        buf.push('<option value="' + version + '">' + title + '</option>');
-      }
-    });
-
-    buf.push('</select>');
-    return buf.join('');
-  }
-
-  function build_doctype_select(current_doctype) {
-    var buf = ['<select>'];
-
-    $.each(all_doctypes, function(doctype, title) {
-      if (doctype == current_doctype)
-        buf.push('<option value="' + doctype + '" selected="selected">' +
-                 all_doctypes[current_doctype] + '</option>');
-      else
-        buf.push('<option value="' + doctype + '">' + title + '</option>');
-    });
-    if (!(current_doctype in all_doctypes)) {
-        // In case we're browsing a doctype that is not yet in all_doctypes.
-        buf.push('<option value="' + current_doctype + '" selected="selected">' +
-                 current_doctype + '</option>');
-        all_doctypes[current_doctype] = current_doctype;
-    }
-    buf.push('</select>');
-    return buf.join('');
-  }
-
-  function navigate_to_first_existing(urls) {
-    // Navigate to the first existing URL in urls.
-    var url = urls.shift();
-
-    // Web browsers won't redirect file:// urls to file urls using ajax but
-    // its useful for local testing
-    if (url.startsWith("file://")) {
-      window.location.href = url;
-      return;
-    }
-
-    if (urls.length == 0) {
-      window.location.href = url;
-      return;
-    }
-    $.ajax({
-      url: url,
-      success: function() {
-        window.location.href = url;
-      },
-      error: function() {
-        navigate_to_first_existing(urls);
-      }
-    });
-  }
-
-  function get_docroot_url() {
-    var url = window.location.href;
-    var root = DOCUMENTATION_OPTIONS.URL_ROOT;
-
-    var urlarray = url.split('/');
-    // Trim off anything after '/'
-    urlarray.pop();
-    var depth = (root.match(/\.\.\//g) || []).length;
-    for (var i = 0; i < depth; i++) {
-      urlarray.pop();
-    }
-
-    return urlarray.join('/') + '/';
-  }
-
-  function on_version_switch() {
-    var selected_version = $(this).children('option:selected').attr('value');
-    var url = window.location.href;
-    var current_version = DOCUMENTATION_OPTIONS.VERSION;
-    var docroot = get_docroot_url()
-
-    var new_versionpath = selected_version + '/';
-    if (selected_version == "dev")
-        new_versionpath = '';
-
-    // dev versions have no version prefix
-    if (current_version == "dev") {
-        var new_url = docroot + new_versionpath + url.replace(docroot, "");
-        var fallback_url = docroot + new_versionpath;
-    } else {
-        var new_url = url.replace('/' + current_version + '/', '/' + new_versionpath);
-        var fallback_url = new_url.replace(url.replace(docroot, ""), "");
-    }
-
-    console.log(get_docroot_url())
-    console.log(url + " to url " + new_url);
-    console.log(url + " to fallback " + fallback_url);
-
-    if (new_url != url) {
-      navigate_to_first_existing([
-        new_url,
-        fallback_url,
-        'https://www.yoctoproject.org/docs/',
-      ]);
-    }
-  }
-
-  function on_doctype_switch() {
-    var selected_doctype = $(this).children('option:selected').attr('value');
-    var url = window.location.href;
-    if (selected_doctype == 'mega') {
-      var docroot = get_docroot_url()
-      var current_version = DOCUMENTATION_OPTIONS.VERSION;
-      // Assume manuals before 3.2 are using old docbook mega-manual
-      if (ver_compare(current_version, "3.2") < 0) {
-        var new_url = docroot + "mega-manual/mega-manual.html";
-      } else {
-        var new_url = docroot + "singleindex.html";
-      }
-    } else {
-      var new_url = url.replace("singleindex.html", "index.html")
-    }
-
-    if (new_url != url) {
-      navigate_to_first_existing([
-        new_url,
-        'https://www.yoctoproject.org/docs/',
-      ]);
-    }
-  }
-
-  // Returns the current doctype based upon the url
-  function doctype_segment_from_url(url) {
-    if (url.includes("singleindex") || url.includes("mega-manual"))
-      return "mega";
-    return "single";
-  }
-
-  $(document).ready(function() {
-    var release = DOCUMENTATION_OPTIONS.VERSION;
-    var current_doctype = doctype_segment_from_url(window.location.href);
-    var current_series = release.substr(0, 3);
-    var version_select = build_version_select(current_series, release);
-
-    $('.version_switcher_placeholder').html(version_select);
-    $('.version_switcher_placeholder select').bind('change', on_version_switch);
-
-    var doctype_select = build_doctype_select(current_doctype);
-
-    $('.doctype_switcher_placeholder').html(doctype_select);
-    $('.doctype_switcher_placeholder select').bind('change', on_doctype_switch);
-
-    if (ver_compare(release, "3.1") < 0) {
-      $('#outdated-warning').html('Version ' + release + ' of the project is now considered obsolete, please select and use a more recent version');
-      $('#outdated-warning').css('padding', '.5em');
-    } else if (release != "dev") {
-      $.each(all_versions, function(version, title) {
-        var series = version.substr(0, 3);
-        if (series == current_series && version != release) {
-          $('#outdated-warning').html('This document is for outdated version ' + release + ', you should select the latest release version in this series, ' + version + '.');
-          $('#outdated-warning').css('padding', '.5em');
-        }
-      });
-    }
-  });
-})();

bitbake/doc/sphinx-static/theme_overrides.css

@@ -1,162 +0,0 @@
-/*
-    SPDX-License-Identifier: CC-BY-2.0-UK
-*/
-
-body {
-  font-family: Verdana, Sans, sans-serif;
-  margin:  0em auto;
-  color: #333;
-}
-
-h1,h2,h3,h4,h5,h6,h7 {
-  font-family: Arial, Sans;
-  color: #00557D;
-  clear: both;
-}
-
-h1 {
-  font-size: 2em;
-  text-align: left;
-  padding: 0em 0em 0em 0em;
-  margin: 2em 0em 0em 0em;
-}
-
-h2.subtitle {
-  margin: 0.10em 0em 3.0em 0em;
-  padding: 0em 0em 0em 0em;
-  font-size: 1.8em;
-  padding-left: 20%;
-  font-weight: normal;
-  font-style: italic;
-}
-
-h2 {
-  margin: 2em 0em 0.66em 0em;
-  padding: 0.5em 0em 0em 0em;
-  font-size: 1.5em;
-  font-weight: bold;
-}
-
-h3.subtitle {
-  margin: 0em 0em 1em 0em;
-  padding: 0em 0em 0em 0em;
-  font-size: 142.14%;
-  text-align: right;
-}
-
-h3 {
-  margin: 1em 0em 0.5em 0em;
-  padding: 1em 0em 0em 0em;
-  font-size: 140%;
-  font-weight: bold;
-}
-
-h4 {
-  margin: 1em 0em 0.5em 0em;
-  padding: 1em 0em 0em 0em;
-  font-size: 120%;
-  font-weight: bold;
-}
-
-h5 {
-  margin: 1em 0em 0.5em 0em;
-  padding: 1em 0em 0em 0em;
-  font-size: 110%;
-  font-weight: bold;
-}
-
-h6 {
-  margin: 1em 0em 0em 0em;
-  padding: 1em 0em 0em 0em;
-  font-size: 110%;
-  font-weight: bold;
-}
-
-em {
-  font-weight: bold;
-}
-
-.pre {
-  font-size: medium;
-  font-family: Courier, monospace;
-}
-
-.wy-nav-content a {
-  text-decoration: underline;
-  color: #444;
-  background: transparent;
-}
-
-.wy-nav-content a:hover {
-  text-decoration: underline;
-  background-color: #dedede;
-}
-
-.wy-nav-content a:visited {
-  color: #444;
-}
-
-[alt='Permalink'] { color: #eee; }
-[alt='Permalink']:hover { color: black; }
-
-@media screen {
-    /* content column
-     *
-     * RTD theme's default is 800px as max width for the content, but we have
-     * tables with tons of columns, which need the full width of the view-port.
-     */
-
-    .wy-nav-content{max-width: none; }
-
-    /* inline literal: drop the borderbox, padding and red color */
-    code, .rst-content tt, .rst-content code {
-        color: inherit;
-        border: none;
-        padding: unset;
-        background: inherit;
-        font-size: 85%;
-    }
-
-    .rst-content tt.literal,.rst-content tt.literal,.rst-content code.literal {
-        color: inherit;
-    }
-
-    /* Admonition should be gray, not blue or green */
-    .rst-content .note .admonition-title,
-    .rst-content .tip .admonition-title,
-    .rst-content .warning .admonition-title,
-    .rst-content .caution .admonition-title,
-    .rst-content .important .admonition-title {
-        background: #f0f0f2;
-        color: #00557D;
-
-    }
-
-    .rst-content .note,
-    .rst-content .tip,
-    .rst-content .important,
-    .rst-content .warning,
-    .rst-content .caution  {
-        background: #f0f0f2;
-    }
-
-    /* Remove the icon in front of note/tip element, and before the logo */
-    .icon-home:before, .rst-content .admonition-title:before {
-        display: none
-    }
-
-    /* a custom informalexample container is used in some doc */
-    .informalexample {
-        border: 1px solid;
-        border-color: #aaa;
-        margin: 1em 0em;
-        padding: 1em;
-        page-break-inside: avoid;
-    }
-
-    /* Remove the blue background in the top left corner, around the logo */
-    .wy-side-nav-search {
-        background: inherit;
-    }
-
-}

bitbake/doc/template/component.title.xsl

@@ -0,0 +1,39 @@
+<xsl:stylesheet version="1.0"
+  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
+  xmlns:d="http://docbook.org/ns/docbook"
+  xmlns="http://www.w3.org/1999/xhtml"
+  exclude-result-prefixes="d">
+  
+  <xsl:template name="component.title">
+    <xsl:param name="node" select="."/>
+    
+    <xsl:variable name="level">
+      <xsl:choose>
+        <xsl:when test="ancestor::d:section">
+          <xsl:value-of select="count(ancestor::d:section)+1"/>
+        </xsl:when>
+        <xsl:when test="ancestor::d:sect5">6</xsl:when>
+        <xsl:when test="ancestor::d:sect4">5</xsl:when>
+        <xsl:when test="ancestor::d:sect3">4</xsl:when>
+        <xsl:when test="ancestor::d:sect2">3</xsl:when>
+        <xsl:when test="ancestor::d:sect1">2</xsl:when>
+        <xsl:otherwise>1</xsl:otherwise>
+      </xsl:choose>
+    </xsl:variable>
+    <xsl:element name="h{$level+1}" namespace="http://www.w3.org/1999/xhtml">
+      <xsl:attribute name="class">title</xsl:attribute>
+      <xsl:if test="$generate.id.attributes = 0">
+        <xsl:call-template name="anchor">
+          <xsl:with-param name="node" select="$node"/>
+          <xsl:with-param name="conditional" select="0"/>
+        </xsl:call-template>
+      </xsl:if>
+      <xsl:apply-templates select="$node" mode="object.title.markup">
+        <xsl:with-param name="allow-anchors" select="1"/>
+      </xsl:apply-templates>
+      <xsl:call-template name="permalink">
+        <xsl:with-param name="node" select="$node"/>
+      </xsl:call-template>
+    </xsl:element>
+  </xsl:template>
+</xsl:stylesheet>

bitbake/doc/template/db-pdf.xsl

@@ -0,0 +1,64 @@
+<?xml version='1.0'?>
+<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" xmlns:fo="http://www.w3.org/1999/XSL/Format" version="1.0">
+  
+  <xsl:import href="http://docbook.sourceforge.net/release/xsl/current/fo/docbook.xsl" />
+
+  <!-- check project-plan.sh for how this is generated, needed to tweak 
+       the cover page     
+    -->
+  <xsl:include href="/tmp/titlepage.xsl"/> 
+
+  <!-- To force a page break in document, i.e per section add a 
+      <?hard-pagebreak?> tag.
+  -->
+ <xsl:template match="processing-instruction('hard-pagebreak')">
+   <fo:block break-before='page' />
+ </xsl:template>
+
+  <!--Fix for defualt indent getting TOC all wierd..
+      See http://sources.redhat.com/ml/docbook-apps/2005-q1/msg00455.html 
+      FIXME: must be a better fix
+    -->
+  <xsl:param name="body.start.indent" select="'0'"/>
+  <!--<xsl:param name="title.margin.left" select="'0'"/>-->
+
+  <!-- stop long-ish header titles getting wrapped -->
+  <xsl:param name="header.column.widths">1 10 1</xsl:param>
+
+  <!-- customise headers and footers a little --> 
+
+  <xsl:template name="head.sep.rule">
+   <xsl:if test="$header.rule != 0">
+     <xsl:attribute name="border-bottom-width">0.5pt</xsl:attribute>
+     <xsl:attribute name="border-bottom-style">solid</xsl:attribute>
+     <xsl:attribute name="border-bottom-color">#cccccc</xsl:attribute>
+   </xsl:if>
+  </xsl:template>
+
+  <xsl:template name="foot.sep.rule">
+    <xsl:if test="$footer.rule != 0">
+     <xsl:attribute name="border-top-width">0.5pt</xsl:attribute>
+     <xsl:attribute name="border-top-style">solid</xsl:attribute>
+     <xsl:attribute name="border-top-color">#cccccc</xsl:attribute>
+    </xsl:if>
+  </xsl:template>
+
+  <xsl:attribute-set name="header.content.properties">
+    <xsl:attribute name="color">#cccccc</xsl:attribute>
+  </xsl:attribute-set>
+
+  <xsl:attribute-set name="footer.content.properties">
+    <xsl:attribute name="color">#cccccc</xsl:attribute>
+  </xsl:attribute-set>
+
+ 
+  <!-- general settings -->
+
+  <xsl:param name="fop1.extensions" select="1"></xsl:param>
+  <xsl:param name="paper.type" select="'A4'"></xsl:param>
+  <xsl:param name="section.autolabel" select="1"></xsl:param>
+  <xsl:param name="body.font.family" select="'verasans'"></xsl:param>
+  <xsl:param name="title.font.family" select="'verasans'"></xsl:param>
+  <xsl:param name="monospace.font.family" select="'veramono'"></xsl:param>
+
+</xsl:stylesheet>

bitbake/doc/template/division.title.xsl

@@ -0,0 +1,25 @@
+<xsl:stylesheet version="1.0"
+  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
+  xmlns:d="http://docbook.org/ns/docbook"
+  xmlns="http://www.w3.org/1999/xhtml"
+  exclude-result-prefixes="d">
+  
+  <xsl:template name="division.title">
+    <xsl:param name="node" select="."/>
+    
+    <h1>
+      <xsl:attribute name="class">title</xsl:attribute>
+      <xsl:call-template name="anchor">
+        <xsl:with-param name="node" select="$node"/>
+        <xsl:with-param name="conditional" select="0"/>
+      </xsl:call-template>
+      <xsl:apply-templates select="$node" mode="object.title.markup">
+        <xsl:with-param name="allow-anchors" select="1"/>
+      </xsl:apply-templates>
+      <xsl:call-template name="permalink">
+        <xsl:with-param name="node" select="$node"/>
+      </xsl:call-template>
+    </h1>
+  </xsl:template>
+</xsl:stylesheet>
+

bitbake/doc/template/fop-config.xml

@@ -0,0 +1,58 @@
+<fop version="1.0">
+
+  <!-- Strict user configuration -->
+  <strict-configuration>true</strict-configuration>
+
+  <!-- Strict FO validation -->
+  <strict-validation>true</strict-validation>
+
+   <!--
+    Set the baseDir so common/openedhand.svg references in plans still
+    work ok. Note, relative file references to current dir should still work.
+    -->	
+  <base>../template</base>
+  <font-base>../template</font-base>
+ 
+  <!-- Source resolution in dpi (dots/pixels per inch) for determining the
+       size of pixels in SVG and bitmap images, default: 72dpi -->
+  <!-- <source-resolution>72</source-resolution> -->
+  <!-- Target resolution in dpi (dots/pixels per inch) for specifying the
+       target resolution for generated bitmaps, default: 72dpi -->
+  <!-- <target-resolution>72</target-resolution> -->
+ 
+  <!-- default page-height and page-width, in case
+       value is specified as auto -->
+  <default-page-settings height="11in" width="8.26in"/> 
+ 
+  <!-- <use-cache>false</use-cache> -->
+ 
+  <renderers>
+    <renderer mime="application/pdf">
+      <fonts>
+        <font  metrics-file="VeraMono.xml"
+               kerning="yes" 
+               embed-url="VeraMono.ttf">
+          <font-triplet name="veramono" style="normal" weight="normal"/>
+        </font>
+
+        <font  metrics-file="VeraMoBd.xml"
+               kerning="yes" 
+               embed-url="VeraMoBd.ttf">
+          <font-triplet name="veramono" style="normal" weight="bold"/>
+        </font>
+
+        <font  metrics-file="Vera.xml"
+               kerning="yes" 
+               embed-url="Vera.ttf">
+          <font-triplet name="verasans" style="normal" weight="normal"/>
+          <font-triplet name="verasans" style="normal" weight="bold"/>
+          <font-triplet name="verasans" style="italic" weight="normal"/>
+          <font-triplet name="verasans" style="italic" weight="bold"/>
+        </font>
+        
+        <auto-detect/>
+      </fonts>
+    </renderer>
+  </renderers>
+</fop>
+

bitbake/doc/template/formal.object.heading.xsl

@@ -0,0 +1,21 @@
+<xsl:stylesheet version="1.0"
+  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
+  xmlns:d="http://docbook.org/ns/docbook"
+  xmlns="http://www.w3.org/1999/xhtml"
+  exclude-result-prefixes="d">
+  
+  <xsl:template name="formal.object.heading">
+    <xsl:param name="object" select="."/>
+    <xsl:param name="title">
+      <xsl:apply-templates select="$object" mode="object.title.markup">
+        <xsl:with-param name="allow-anchors" select="1"/>
+      </xsl:apply-templates>
+    </xsl:param>
+    <p class="title">
+      <b><xsl:copy-of select="$title"/></b>
+      <xsl:call-template name="permalink">
+        <xsl:with-param name="node" select="$object"/>
+      </xsl:call-template>
+    </p>
+  </xsl:template>
+</xsl:stylesheet>

bitbake/doc/template/gloss-permalinks.xsl

@@ -0,0 +1,14 @@
+<xsl:stylesheet version="1.0"
+  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
+  xmlns:d="http://docbook.org/ns/docbook"
+  xmlns="http://www.w3.org/1999/xhtml">
+
+  <xsl:template match="glossentry/glossterm">
+    <xsl:apply-imports/>
+    <xsl:if test="$generate.permalink != 0">
+      <xsl:call-template name="permalink">
+        <xsl:with-param name="node" select=".."/>
+      </xsl:call-template>
+    </xsl:if>
+  </xsl:template>
+</xsl:stylesheet>

bitbake/doc/template/permalinks.xsl

@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<xsl:stylesheet version="1.0"
+  xmlns="http://www.w3.org/1999/xhtml"
+  xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
+
+  <xsl:param name="generate.permalink" select="1"/>
+  <xsl:param name="permalink.text">¶</xsl:param>
+
+  <xsl:template name="permalink">
+    <xsl:param name="node"/>
+
+    <xsl:if test="$generate.permalink != '0'">
+      <span class="permalink">
+        <a alt="Permalink" title="Permalink">
+          <xsl:attribute name="href">
+            <xsl:call-template name="href.target">
+              <xsl:with-param name="object"  select="$node"/>
+            </xsl:call-template>
+          </xsl:attribute>
+          <xsl:copy-of select="$permalink.text"/>
+        </a>
+      </span>
+    </xsl:if>
+  </xsl:template>
+</xsl:stylesheet>

bitbake/doc/template/section.title.xsl

@@ -0,0 +1,55 @@
+<xsl:stylesheet version="1.0"
+  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
+  xmlns:d="http://docbook.org/ns/docbook"
+  xmlns="http://www.w3.org/1999/xhtml" exclude-result-prefixes="d">
+
+  <xsl:template name="section.title">
+    <xsl:variable name="section"
+      select="(ancestor::section |
+               ancestor::simplesect|
+               ancestor::sect1|
+               ancestor::sect2|
+               ancestor::sect3|
+               ancestor::sect4|
+               ancestor::sect5)[last()]"/>
+
+    <xsl:variable name="renderas">
+      <xsl:choose>
+        <xsl:when test="$section/@renderas = 'sect1'">1</xsl:when>
+        <xsl:when test="$section/@renderas = 'sect2'">2</xsl:when>
+        <xsl:when test="$section/@renderas = 'sect3'">3</xsl:when>
+        <xsl:when test="$section/@renderas = 'sect4'">4</xsl:when>
+        <xsl:when test="$section/@renderas = 'sect5'">5</xsl:when>
+        <xsl:otherwise><xsl:value-of select="''"/></xsl:otherwise>
+      </xsl:choose>
+    </xsl:variable>
+
+    <xsl:variable name="level">
+      <xsl:choose>
+        <xsl:when test="$renderas != ''">
+          <xsl:value-of select="$renderas"/>
+        </xsl:when>
+        <xsl:otherwise>
+          <xsl:call-template name="section.level">
+            <xsl:with-param name="node" select="$section"/>
+          </xsl:call-template>
+        </xsl:otherwise>
+      </xsl:choose>
+    </xsl:variable>
+
+    <xsl:call-template name="section.heading">
+      <xsl:with-param name="section" select="$section"/>
+      <xsl:with-param name="level" select="$level"/>
+      <xsl:with-param name="title">
+        <xsl:apply-templates select="$section" mode="object.title.markup">
+          <xsl:with-param name="allow-anchors" select="1"/>
+        </xsl:apply-templates>
+        <xsl:if test="$level &gt; 0">
+          <xsl:call-template name="permalink">
+            <xsl:with-param name="node" select="$section"/>
+          </xsl:call-template>
+        </xsl:if>
+      </xsl:with-param>
+    </xsl:call-template>
+  </xsl:template>
+</xsl:stylesheet>

bitbake/doc/tools/docbook-to-pdf

@@ -0,0 +1,51 @@
+#!/bin/sh
+	
+if [ -z "$1" -o -z "$2" ]; then
+   echo "usage: [-v] $0 <docbook file> <templatedir>"
+   echo
+   echo "*NOTE* you need xsltproc, fop and nwalsh docbook stylesheets" 
+   echo "       installed for this to work!"
+   echo
+   exit 0
+fi
+
+FO=`echo $1 | sed s/.xml/.fo/` || exit 1
+PDF=`echo $1 | sed s/.xml/.pdf/` || exit 1
+TEMPLATEDIR=$2
+
+##
+# These URI should be rewritten by your distribution's xml catalog to
+# match your localy installed XSL stylesheets.
+XSL_BASE_URI="http://docbook.sourceforge.net/release/xsl/current"
+
+# Creates a temporary XSL stylesheet based on titlepage.xsl
+xsltproc -o /tmp/titlepage.xsl                                           \
+	 --xinclude                                                      \
+         $XSL_BASE_URI/template/titlepage.xsl \
+         $TEMPLATEDIR/titlepage.templates.xml || exit 1
+
+# Creates the file needed for FOP
+xsltproc --xinclude                    \
+	 --stringparam hyphenate false \
+	 --stringparam formal.title.placement "figure after" \
+	 --stringparam ulink.show 1 \
+         --stringparam  body.font.master  9 \
+         --stringparam  title.font.master  11 \
+         --stringparam draft.watermark.image "$TEMPLATEDIR/draft.png" \
+         --stringparam  chapter.autolabel 1 \
+         --stringparam  appendix.autolabel A \
+         --stringparam  section.autolabel 1 \
+         --stringparam  section.label.includes.component.label 1 \
+         --output $FO               \
+         $TEMPLATEDIR/db-pdf.xsl    \
+	 $1                 || exit 1
+
+# Invokes the Java version of FOP.  Uses the additional configuration file common/fop-config.xml
+fop -c $TEMPLATEDIR/fop-config.xml -fo $FO -pdf $PDF       || exit 1
+
+rm -f $FO
+rm -f  /tmp/titlepage.xsl
+
+echo
+echo " #### Success! $PDF ready. ####"
+echo

bitbake/lib/bb/COW.py

@@ -1,8 +1,23 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
 # This is a copy on write dictionary and set which abuses classes to try and be nice and fast.
 #
 # Copyright (C) 2006 Tim Ansell
 #
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
 #Please Note:
 # Be careful when using mutable types (ie Dict and Lists) - operations involving these are SLOW.
 # Assign a file to __warn__ to get warnings about slow operations.
@@ -10,6 +25,7 @@
 
 
 import copy
+import types
 ImmutableTypes = (
     bool,
     complex,
@@ -134,7 +150,7 @@ class COWDictMeta(COWMeta):
                 yield value
             if type == "items":
                 yield (key, value)
-        return
+        raise StopIteration()
 
     def iterkeys(cls):
         return cls.iter("keys")

bitbake/lib/bb/__init__.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
 # BitBake Build System Python Library
 #
@@ -6,22 +8,25 @@
 #
 # Based on Gentoo's portage.py.
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
-__version__ = "1.46.0"
+__version__ = "1.36.0"
 
 import sys
-if sys.version_info < (3, 5, 0):
-    raise RuntimeError("Sorry, python 3.5.0 or later is required for this version of bitbake")
+if sys.version_info < (3, 4, 0):
+    raise RuntimeError("Sorry, python 3.4.0 or later is required for this version of bitbake")
 
-if sys.version_info < (3, 10, 0):
-    # With python 3.8 and 3.9, we see errors of "libgcc_s.so.1 must be installed for pthread_cancel to work"
-    # https://stackoverflow.com/questions/64797838/libgcc-s-so-1-must-be-installed-for-pthread-cancel-to-work
-    # https://bugs.ams1.psf.io/issue42888
-    # so ensure libgcc_s is loaded early on
-    import ctypes
-    libgcc_s = ctypes.CDLL('libgcc_s.so.1')
 
 class BBHandledException(Exception):
     """
@@ -50,13 +55,7 @@ class BBLogger(Logger):
         Logger.__init__(self, name)
 
     def bbdebug(self, level, msg, *args, **kwargs):
-        loglevel = logging.DEBUG - level + 1
-        if not bb.event.worker_pid:
-            if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb.msg.loggerDefaultDomains[self.name]):
-                return
-            if loglevel < bb.msg.loggerDefaultLogLevel:
-                return
-        return self.log(loglevel, msg, *args, **kwargs)
+        return self.log(logging.DEBUG - level + 1, msg, *args, **kwargs)
 
     def plain(self, msg, *args, **kwargs):
         return self.log(logging.INFO + 1, msg, *args, **kwargs)
@@ -64,10 +63,6 @@ class BBLogger(Logger):
     def verbose(self, msg, *args, **kwargs):
         return self.log(logging.INFO - 1, msg, *args, **kwargs)
 
-    def verbnote(self, msg, *args, **kwargs):
-        return self.log(logging.INFO + 2, msg, *args, **kwargs)
-
-
 logging.raiseExceptions = False
 logging.setLoggerClass(BBLogger)
 
@@ -98,18 +93,6 @@ def debug(lvl, *args):
 def note(*args):
     mainlogger.info(''.join(args))
 
-#
-# A higher prioity note which will show on the console but isn't a warning
-#
-# Something is happening the user should be aware of but they probably did
-# something to make it happen
-#
-def verbnote(*args):
-    mainlogger.verbnote(''.join(args))
-
-#
-# Warnings - things the user likely needs to pay attention to and fix
-#
 def warn(*args):
     mainlogger.warning(''.join(args))
 

bitbake/lib/bb/build.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
 # BitBake 'Build' implementation
 #
@@ -8,13 +10,25 @@
 #
 # Based on Gentoo's portage.py.
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 #
 # Based on functions from the base bb module, Copyright 2003 Holger Schurig
 
 import os
 import sys
 import logging
+import shlex
 import glob
 import time
 import stat
@@ -27,8 +41,7 @@ from bb import data, event, utils
 bblogger = logging.getLogger('BitBake')
 logger = logging.getLogger('BitBake.Build')
 
-verboseShellLogging = False
-verboseStdoutLogging = False
+NULL = open(os.devnull, 'r+')
 
 __mtime_cache = {}
 
@@ -56,20 +69,34 @@ else:
 builtins['bb'] = bb
 builtins['os'] = os
 
+class FuncFailed(Exception):
+    def __init__(self, name = None, logfile = None):
+        self.logfile = logfile
+        self.name = name
+        if name:
+            self.msg = 'Function failed: %s' % name
+        else:
+            self.msg = "Function failed"
+
+    def __str__(self):
+        if self.logfile and os.path.exists(self.logfile):
+            msg = ("%s (log file is located at %s)" %
+                   (self.msg, self.logfile))
+        else:
+            msg = self.msg
+        return msg
+
 class TaskBase(event.Event):
     """Base class for task events"""
 
-    def __init__(self, t, fn, logfile, d):
+    def __init__(self, t, logfile, d):
         self._task = t
-        self._fn = fn
         self._package = d.getVar("PF")
         self._mc = d.getVar("BB_CURRENT_MC")
         self.taskfile = d.getVar("FILE")
         self.taskname = self._task
         self.logfile = logfile
         self.time = time.time()
-        self.pn = d.getVar("PN")
-        self.pv = d.getVar("PV")
         event.Event.__init__(self)
         self._message = "recipe %s: task %s: %s" % (d.getVar("PF"), t, self.getDisplayName())
 
@@ -86,8 +113,8 @@ class TaskBase(event.Event):
 
 class TaskStarted(TaskBase):
     """Task execution started"""
-    def __init__(self, t, fn, logfile, taskflags, d):
-        super(TaskStarted, self).__init__(t, fn, logfile, d)
+    def __init__(self, t, logfile, taskflags, d):
+        super(TaskStarted, self).__init__(t, logfile, d)
         self.taskflags = taskflags
 
 class TaskSucceeded(TaskBase):
@@ -96,9 +123,9 @@ class TaskSucceeded(TaskBase):
 class TaskFailed(TaskBase):
     """Task execution failed"""
 
-    def __init__(self, task, fn, logfile, metadata, errprinted = False):
+    def __init__(self, task, logfile, metadata, errprinted = False):
         self.errprinted = errprinted
-        super(TaskFailed, self).__init__(task, fn, logfile, metadata)
+        super(TaskFailed, self).__init__(task, logfile, metadata)
 
 class TaskFailedSilent(TaskBase):
     """Task execution failed (silently)"""
@@ -108,8 +135,8 @@ class TaskFailedSilent(TaskBase):
 
 class TaskInvalid(TaskBase):
 
-    def __init__(self, task, fn, metadata):
-        super(TaskInvalid, self).__init__(task, fn, None, metadata)
+    def __init__(self, task, metadata):
+        super(TaskInvalid, self).__init__(task, None, metadata)
         self._message = "No such task '%s'" % task
 
 class TaskProgress(event.Event):
@@ -151,33 +178,15 @@ class LogTee(object):
 
     def __repr__(self):
         return '<LogTee {0}>'.format(self.name)
-
     def flush(self):
         self.outfile.flush()
 
-
-class StdoutNoopContextManager:
-    """
-    This class acts like sys.stdout, but adds noop __enter__ and __exit__ methods.
-    """
-    def __enter__(self):
-        return sys.stdout
-
-    def __exit__(self, *exc_info):
-        pass
-
-    def write(self, string):
-        return sys.stdout.write(string)
-
-    def flush(self):
-        sys.stdout.flush()
-
-    @property
-    def name(self):
-        return sys.stdout.name
-
-
-def exec_func(func, d, dirs = None):
+#
+# pythonexception allows the python exceptions generated to be raised
+# as the real exceptions (not FuncFailed) and without a backtrace at the 
+# origin of the failure.
+#
+def exec_func(func, d, dirs = None, pythonexception=False):
     """Execute a BB 'function'"""
 
     try:
@@ -249,7 +258,7 @@ def exec_func(func, d, dirs = None):
 
     with bb.utils.fileslocked(lockfiles):
         if ispython:
-            exec_func_python(func, d, runfile, cwd=adir)
+            exec_func_python(func, d, runfile, cwd=adir, pythonexception=pythonexception)
         else:
             exec_func_shell(func, d, runfile, cwd=adir)
 
@@ -269,7 +278,7 @@ _functionfmt = """
 {function}(d)
 """
 logformatter = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
-def exec_func_python(func, d, runfile, cwd=None):
+def exec_func_python(func, d, runfile, cwd=None, pythonexception=False):
     """Execute a python BB 'function'"""
 
     code = _functionfmt.format(function=func)
@@ -293,8 +302,14 @@ def exec_func_python(func, d, runfile, cwd=None):
         lineno = int(d.getVarFlag(func, "lineno", False))
         bb.methodpool.insert_method(func, text, fn, lineno - 1)
 
-        comp = utils.better_compile(code, func, "exec_func_python() autogenerated")
-        utils.better_exec(comp, {"d": d}, code, "exec_func_python() autogenerated")
+        comp = utils.better_compile(code, func, "exec_python_func() autogenerated")
+        utils.better_exec(comp, {"d": d}, code, "exec_python_func() autogenerated", pythonexception=pythonexception)
+    except (bb.parse.SkipRecipe, bb.build.FuncFailed):
+        raise
+    except:
+        if pythonexception:
+            raise
+        raise FuncFailed(func, None)
     finally:
         bb.debug(2, "Python function %s finished" % func)
 
@@ -322,42 +337,6 @@ trap 'bb_exit_handler' 0
 set -e
 '''
 
-def create_progress_handler(func, progress, logfile, d):
-    if progress == 'percent':
-        # Use default regex
-        return bb.progress.BasicProgressHandler(d, outfile=logfile)
-    elif progress.startswith('percent:'):
-        # Use specified regex
-        return bb.progress.BasicProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
-    elif progress.startswith('outof:'):
-        # Use specified regex
-        return bb.progress.OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
-    elif progress.startswith("custom:"):
-        # Use a custom progress handler that was injected via OE_EXTRA_IMPORTS or __builtins__
-        import functools
-        from types import ModuleType
-
-        parts = progress.split(":", 2)
-        _, cls, otherargs = parts[0], parts[1], (parts[2] or None) if parts[2:] else None
-        if cls:
-            def resolve(x, y):
-                if not x:
-                    return None
-                if isinstance(x, ModuleType):
-                    return getattr(x, y, None)
-                return x.get(y)
-            cls_obj = functools.reduce(resolve, cls.split("."), bb.utils._context)
-            if not cls_obj:
-                # Fall-back on __builtins__
-                cls_obj = functools.reduce(lambda x, y: x.get(y), cls.split("."), __builtins__)
-            if cls_obj:
-                return cls_obj(d, outfile=logfile, otherargs=otherargs)
-            bb.warn('%s: unknown custom progress handler in task progress varflag value "%s", ignoring' % (func, cls))
-    else:
-        bb.warn('%s: invalid task progress varflag value "%s", ignoring' % (func, progress))
-
-    return logfile
-
 def exec_func_shell(func, d, runfile, cwd=None):
     """Execute a shell function from the metadata
 
@@ -374,7 +353,7 @@ def exec_func_shell(func, d, runfile, cwd=None):
 
         bb.data.emit_func(func, script, d)
 
-        if verboseShellLogging or bb.utils.to_boolean(d.getVar("BB_VERBOSE_LOGS", False)):
+        if bb.msg.loggerVerboseLogs:
             script.write("set -x\n")
         if cwd:
             script.write("cd '%s'\n" % cwd)
@@ -394,14 +373,24 @@ exit $ret
         if fakerootcmd:
             cmd = [fakerootcmd, runfile]
 
-    if verboseStdoutLogging:
-        logfile = LogTee(logger, StdoutNoopContextManager())
+    if bb.msg.loggerDefaultVerbose:
+        logfile = LogTee(logger, sys.stdout)
     else:
-        logfile = StdoutNoopContextManager()
+        logfile = sys.stdout
 
     progress = d.getVarFlag(func, 'progress')
     if progress:
-        logfile = create_progress_handler(func, progress, logfile, d)
+        if progress == 'percent':
+            # Use default regex
+            logfile = bb.progress.BasicProgressHandler(d, outfile=logfile)
+        elif progress.startswith('percent:'):
+            # Use specified regex
+            logfile = bb.progress.BasicProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
+        elif progress.startswith('outof:'):
+            # Use specified regex
+            logfile = bb.progress.OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
+        else:
+            bb.warn('%s: invalid task progress varflag value "%s", ignoring' % (func, progress))
 
     fifobuffer = bytearray()
     def readfifo(data):
@@ -420,8 +409,6 @@ exit $ret
                     bb.plain(value)
                 elif cmd == 'bbnote':
                     bb.note(value)
-                elif cmd == 'bbverbnote':
-                    bb.verbnote(value)
                 elif cmd == 'bbwarn':
                     bb.warn(value)
                 elif cmd == 'bberror':
@@ -451,8 +438,13 @@ exit $ret
     with open(fifopath, 'r+b', buffering=0) as fifo:
         try:
             bb.debug(2, "Executing shell function %s" % func)
-            with open(os.devnull, 'r+') as stdin, logfile:
-                bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)])
+
+            try:
+                with open(os.devnull, 'r+') as stdin:
+                    bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)])
+            except bb.process.CmdError:
+                logfn = d.getVar('BB_LOGFILE')
+                raise FuncFailed(func, logfn)
         finally:
             os.unlink(fifopath)
 
@@ -541,6 +533,7 @@ def _exec_task(fn, task, d, quieterr):
                 self.triggered = True
 
     # Handle logfiles
+    si = open('/dev/null', 'r')
     try:
         bb.utils.mkdirhier(os.path.dirname(logfn))
         logfile = open(logfn, 'w')
@@ -554,8 +547,7 @@ def _exec_task(fn, task, d, quieterr):
     ose = [os.dup(sys.stderr.fileno()), sys.stderr.fileno()]
 
     # Replace those fds with our own
-    with open('/dev/null', 'r') as si:
-        os.dup2(si.fileno(), osi[1])
+    os.dup2(si.fileno(), osi[1])
     os.dup2(logfile.fileno(), oso[1])
     os.dup2(logfile.fileno(), ose[1])
 
@@ -577,9 +569,12 @@ def _exec_task(fn, task, d, quieterr):
 
     try:
         try:
-            event.fire(TaskStarted(task, fn, logfn, flags, localdata), localdata)
+            event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
         except (bb.BBHandledException, SystemExit):
             return 1
+        except FuncFailed as exc:
+            logger.error(str(exc))
+            return 1
 
         try:
             for func in (prefuncs or '').split():
@@ -587,20 +582,16 @@ def _exec_task(fn, task, d, quieterr):
             exec_func(task, localdata)
             for func in (postfuncs or '').split():
                 exec_func(func, localdata)
-        except bb.BBHandledException:
-            event.fire(TaskFailed(task, fn, logfn, localdata, True), localdata)
-            return 1
-        except (Exception, SystemExit) as exc:
+        except FuncFailed as exc:
             if quieterr:
-                event.fire(TaskFailedSilent(task, fn, logfn, localdata), localdata)
+                event.fire(TaskFailedSilent(task, logfn, localdata), localdata)
             else:
                 errprinted = errchk.triggered
-                # If the output is already on stdout, we've printed the information in the
-                # logs once already so don't duplicate
-                if verboseStdoutLogging:
-                    errprinted = True
                 logger.error(str(exc))
-                event.fire(TaskFailed(task, fn, logfn, localdata, errprinted), localdata)
+                event.fire(TaskFailed(task, logfn, localdata, errprinted), localdata)
+            return 1
+        except bb.BBHandledException:
+            event.fire(TaskFailed(task, logfn, localdata, True), localdata)
             return 1
     finally:
         sys.stdout.flush()
@@ -617,13 +608,14 @@ def _exec_task(fn, task, d, quieterr):
         os.close(osi[0])
         os.close(oso[0])
         os.close(ose[0])
+        si.close()
 
         logfile.close()
         if os.path.exists(logfn) and os.path.getsize(logfn) == 0:
             logger.debug(2, "Zero size logfn %s, removing", logfn)
             bb.utils.remove(logfn)
             bb.utils.remove(loglink)
-    event.fire(TaskSucceeded(task, fn, logfn, localdata), localdata)
+    event.fire(TaskSucceeded(task, logfn, localdata), localdata)
 
     if not localdata.getVarFlag(task, 'nostamp', False) and not localdata.getVarFlag(task, 'selfstamp', False):
         make_stamp(task, localdata)
@@ -811,7 +803,6 @@ def add_tasks(tasklist, d):
             if name in flags:
                 deptask = d.expand(flags[name])
                 task_deps[name][task] = deptask
-        getTask('mcdepends')
         getTask('depends')
         getTask('rdepends')
         getTask('deptask')
@@ -825,9 +816,6 @@ def add_tasks(tasklist, d):
         task_deps['parents'][task] = []
         if 'deps' in flags:
             for dep in flags['deps']:
-                # Check and warn for "addtask task after foo" while foo does not exist
-                #if not dep in tasklist:
-                #    bb.warn('%s: dependent task %s for %s does not exist' % (d.getVar('PN'), dep, task))
                 dep = d.expand(dep)
                 task_deps['parents'][task].append(dep)
 
@@ -884,12 +872,6 @@ def preceedtask(task, with_recrdeptasks, d):
     that this may lead to the task itself being listed.
     """
     preceed = set()
-
-    # Ignore tasks which don't exist
-    tasks = d.getVar('__BBTASKS', False)
-    if task not in tasks:
-        return preceed
-
     preceed.update(d.getVarFlag(task, 'deps') or [])
     if with_recrdeptasks:
         recrdeptask = d.getVarFlag(task, 'recrdeptask')
@@ -908,8 +890,6 @@ def tasksbetween(task_start, task_end, d):
     def follow_chain(task, endtask, chain=None):
         if not chain:
             chain = []
-        if task in chain:
-            bb.fatal("Circular task dependencies as %s depends on itself via the chain %s" % (task, " -> ".join(chain)))
         chain.append(task)
         for othertask in tasks:
             if othertask == task:

bitbake/lib/bb/cache.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
 # BitBake Cache implementation
 #
@@ -13,19 +15,29 @@
 # Copyright (C) 2005        Holger Hans Peter Freyther
 # Copyright (C) 2005        ROAD GmbH
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import os
+import sys
 import logging
 import pickle
 from collections import defaultdict
 import bb.utils
-import re
 
 logger = logging.getLogger("BitBake.Cache")
 
-__cache_version__ = "152"
+__cache_version__ = "151"
 
 def getCacheFile(path, filename, data_hash):
     return os.path.join(path, filename + "." + data_hash)
@@ -83,21 +95,21 @@ class CoreRecipeInfo(RecipeInfoCommon):
         self.appends = self.listvar('__BBAPPEND', metadata)
         self.nocache = self.getvar('BB_DONT_CACHE', metadata)
 
-        self.provides  = self.depvar('PROVIDES', metadata)
-        self.rprovides = self.depvar('RPROVIDES', metadata)
-        self.pn = self.getvar('PN', metadata) or bb.parse.vars_from_file(filename,metadata)[0]
-        self.packages = self.listvar('PACKAGES', metadata)
-        if not self.packages:
-            self.packages.append(self.pn)
-        self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC', metadata)
-
         self.skipreason = self.getvar('__SKIPPED', metadata)
         if self.skipreason:
+            self.pn = self.getvar('PN', metadata) or bb.parse.BBHandler.vars_from_file(filename,metadata)[0]
             self.skipped = True
+            self.provides  = self.depvar('PROVIDES', metadata)
+            self.rprovides = self.depvar('RPROVIDES', metadata)
             return
 
         self.tasks = metadata.getVar('__BBTASKS', False)
 
+        self.pn = self.getvar('PN', metadata)
+        self.packages = self.listvar('PACKAGES', metadata)
+        if not self.packages:
+            self.packages.append(self.pn)
+
         self.basetaskhashes = self.taskvar('BB_BASEHASH', self.tasks, metadata)
         self.hashfilename = self.getvar('BB_HASHFILENAME', metadata)
 
@@ -113,8 +125,11 @@ class CoreRecipeInfo(RecipeInfoCommon):
         self.stampclean = self.getvar('STAMPCLEAN', metadata)
         self.stamp_extrainfo = self.flaglist('stamp-extra-info', self.tasks, metadata)
         self.file_checksums = self.flaglist('file-checksums', self.tasks, metadata, True)
+        self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC', metadata)
         self.depends          = self.depvar('DEPENDS', metadata)
+        self.provides         = self.depvar('PROVIDES', metadata)
         self.rdepends         = self.depvar('RDEPENDS', metadata)
+        self.rprovides        = self.depvar('RPROVIDES', metadata)
         self.rrecommends      = self.depvar('RRECOMMENDS', metadata)
         self.rprovides_pkg    = self.pkgvar('RPROVIDES', self.packages, metadata)
         self.rdepends_pkg     = self.pkgvar('RDEPENDS', self.packages, metadata)
@@ -208,10 +223,10 @@ class CoreRecipeInfo(RecipeInfoCommon):
 
         # Collect files we may need for possible world-dep
         # calculations
-        if not self.not_world:
+        if self.not_world:
+            logger.debug(1, "EXCLUDE FROM WORLD: %s", fn)
+        else:
             cachedata.possible_world.append(fn)
-        #else:
-        #    logger.debug(2, "EXCLUDE FROM WORLD: %s", fn)
 
         # create a collection of all targets for sanity checking
         # tasks, such as upstream versions, license, and tools for
@@ -220,7 +235,7 @@ class CoreRecipeInfo(RecipeInfoCommon):
 
         cachedata.hashfn[fn] = self.hashfilename
         for task, taskhash in self.basetaskhashes.items():
-            identifier = '%s:%s' % (fn, task)
+            identifier = '%s.%s' % (fn, task)
             cachedata.basetaskhash[identifier] = taskhash
 
         cachedata.inherits[fn] = self.inherits
@@ -234,7 +249,7 @@ def virtualfn2realfn(virtualfn):
     Convert a virtual file name to a real one + the associated subclass keyword
     """
     mc = ""
-    if virtualfn.startswith('mc:'):
+    if virtualfn.startswith('multiconfig:'):
         elems = virtualfn.split(':')
         mc = elems[1]
         virtualfn = ":".join(elems[2:])
@@ -255,7 +270,7 @@ def realfn2virtual(realfn, cls, mc):
     if cls:
         realfn = "virtual:" + cls + ":" + realfn
     if mc:
-        realfn = "mc:" + mc + ":" + realfn
+        realfn = "multiconfig:" + mc + ":" + realfn
     return realfn
 
 def variant2virtual(realfn, variant):
@@ -264,11 +279,11 @@ def variant2virtual(realfn, variant):
     """
     if variant == "":
         return realfn
-    if variant.startswith("mc:"):
+    if variant.startswith("multiconfig:"):
         elems = variant.split(":")
         if elems[2]:
-            return "mc:" + elems[1] + ":virtual:" + ":".join(elems[2:]) + ":" + realfn
-        return "mc:" + elems[1] + ":" + realfn
+            return "multiconfig:" + elems[1] + ":virtual:" + ":".join(elems[2:]) + ":" + realfn
+        return "multiconfig:" + elems[1] + ":" + realfn
     return "virtual:" + variant + ":" + realfn
 
 def parse_recipe(bb_data, bbfile, appends, mc=''):
@@ -346,7 +361,7 @@ class NoCache(object):
             bb_data = self.databuilder.mcdata[mc].createCopy()
             newstores = parse_recipe(bb_data, bbfile, appends, mc)
             for ns in newstores:
-                datastores["mc:%s:%s" % (mc, ns)] = newstores[ns]
+                datastores["multiconfig:%s:%s" % (mc, ns)] = newstores[ns]
 
         return datastores
 
@@ -370,7 +385,6 @@ class Cache(NoCache):
         self.data_fn = None
         self.cacheclean = True
         self.data_hash = data_hash
-        self.filelist_regex = re.compile(r'(?:(?<=:True)|(?<=:False))\s+')
 
         if self.cachedir in [None, '']:
             self.has_cache = False
@@ -381,7 +395,7 @@ class Cache(NoCache):
         self.has_cache = True
         self.cachefile = getCacheFile(self.cachedir, "bb_cache.dat", self.data_hash)
 
-        logger.debug(1, "Cache dir: %s", self.cachedir)
+        logger.debug(1, "Using cache in '%s'", self.cachedir)
         bb.utils.mkdirhier(self.cachedir)
 
         cache_ok = True
@@ -394,17 +408,6 @@ class Cache(NoCache):
             self.load_cachefile()
         elif os.path.isfile(self.cachefile):
             logger.info("Out of date cache found, rebuilding...")
-        else:
-            logger.debug(1, "Cache file %s not found, building..." % self.cachefile)
-
-        # We don't use the symlink, its just for debugging convinience
-        symlink = os.path.join(self.cachedir, "bb_cache.dat")
-        if os.path.exists(symlink):
-            bb.utils.remove(symlink)
-        try:
-            os.symlink(os.path.basename(self.cachefile), symlink)
-        except OSError:
-            pass
 
     def load_cachefile(self):
         cachesize = 0
@@ -421,7 +424,6 @@ class Cache(NoCache):
 
         for cache_class in self.caches_array:
             cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
-            logger.debug(1, 'Loading cache file: %s' % cachefile)
             with open(cachefile, "rb") as cachefile:
                 pickled = pickle.Unpickler(cachefile)
                 # Check cache version information
@@ -609,12 +611,20 @@ class Cache(NoCache):
         if hasattr(info_array[0], 'file_checksums'):
             for _, fl in info_array[0].file_checksums.items():
                 fl = fl.strip()
-                if not fl:
-                    continue
-                # Have to be careful about spaces and colons in filenames
-                flist = self.filelist_regex.split(fl)
-                for f in flist:
-                    if not f or "*" in f:
+                while fl:
+                    # A .split() would be simpler but means spaces or colons in filenames would break
+                    a = fl.find(":True")
+                    b = fl.find(":False")
+                    if ((a < 0) and b) or ((b > 0) and (b < a)):
+                        f = fl[:b+6]
+                        fl = fl[b+7:]
+                    elif ((b < 0) and a) or ((a > 0) and (a < b)):
+                        f = fl[:a+5]
+                        fl = fl[a+6:]
+                    else:
+                        break
+                    fl = fl.strip()
+                    if "*" in f:
                         continue
                     f, exist = f.split(":")
                     if (exist == "True" and not os.path.exists(f)) or (exist == "False" and os.path.exists(f)):
@@ -876,56 +886,3 @@ class MultiProcessCache(object):
             p.dump([data, self.__class__.CACHE_VERSION])
 
         bb.utils.unlockfile(glf)
-
-
-class SimpleCache(object):
-    """
-    BitBake multi-process cache implementation
-
-    Used by the codeparser & file checksum caches
-    """
-
-    def __init__(self, version):
-        self.cachefile = None
-        self.cachedata = None
-        self.cacheversion = version
-
-    def init_cache(self, d, cache_file_name=None, defaultdata=None):
-        cachedir = (d.getVar("PERSISTENT_DIR") or
-                    d.getVar("CACHE"))
-        if not cachedir:
-            return defaultdata
-
-        bb.utils.mkdirhier(cachedir)
-        self.cachefile = os.path.join(cachedir,
-                                      cache_file_name or self.__class__.cache_file_name)
-        logger.debug(1, "Using cache in '%s'", self.cachefile)
-
-        glf = bb.utils.lockfile(self.cachefile + ".lock")
-
-        try:
-            with open(self.cachefile, "rb") as f:
-                p = pickle.Unpickler(f)
-                data, version = p.load()
-        except:
-            bb.utils.unlockfile(glf)
-            return defaultdata
-
-        bb.utils.unlockfile(glf)
-
-        if version != self.cacheversion:
-            return defaultdata
-
-        return data
-
-    def save(self, data):
-        if not self.cachefile:
-            return
-
-        glf = bb.utils.lockfile(self.cachefile + ".lock")
-
-        with open(self.cachefile, "wb") as f:
-            p = pickle.Pickler(f, -1)
-            p.dump([data, self.cacheversion])
-
-        bb.utils.unlockfile(glf)

bitbake/lib/bb/cache_extra.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
 # Extra RecipeInfo will be all defined in this file. Currently,
 # Only Hob (Image Creator) Requests some extra fields. So
@@ -10,8 +12,18 @@
 
 # Copyright (C) 2011, Intel Corporation. All rights reserved.
 
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 from bb.cache import RecipeInfoCommon
 

bitbake/lib/bb/checksum.py

@@ -2,13 +2,24 @@
 #
 # Copyright (C) 2012 Intel Corporation
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import glob
 import operator
 import os
 import stat
+import pickle
 import bb.utils
 import logging
 from bb.cache import MultiProcessCache
@@ -73,7 +84,7 @@ class FileChecksumCache(MultiProcessCache):
             else:
                 dest[0][h] = source[0][h]
 
-    def get_checksums(self, filelist, pn, localdirsexclude):
+    def get_checksums(self, filelist, pn):
         """Get checksums for a list of files"""
 
         def checksum_file(f):
@@ -86,11 +97,8 @@ class FileChecksumCache(MultiProcessCache):
 
         def checksum_dir(pth):
             # Handle directories recursively
-            if pth == "/":
-                bb.fatal("Refusing to checksum /")
             dirchecksums = []
-            for root, dirs, files in os.walk(pth, topdown=True):
-                [dirs.remove(d) for d in list(dirs) if d in localdirsexclude]
+            for root, dirs, files in os.walk(pth):
                 for name in files:
                     fullpth = os.path.join(root, name)
                     checksum = checksum_file(fullpth)

bitbake/lib/bb/codeparser.py

@@ -1,7 +1,3 @@
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
-
 """
 BitBake code parser
 
@@ -25,17 +21,19 @@ import ast
 import sys
 import codegen
 import logging
+import pickle
 import bb.pysh as pysh
+import os.path
 import bb.utils, bb.data
 import hashlib
 from itertools import chain
-from bb.pysh import pyshyacc, pyshlex
+from bb.pysh import pyshyacc, pyshlex, sherrors
 from bb.cache import MultiProcessCache
 
 logger = logging.getLogger('BitBake.CodeParser')
 
 def bbhash(s):
-    return hashlib.sha256(s.encode("utf-8")).hexdigest()
+    return hashlib.md5(s.encode("utf-8")).hexdigest()
 
 def check_indent(codestr):
     """If the code is indented, add a top level piece of code to 'remove' the indentation"""
@@ -56,10 +54,30 @@ def check_indent(codestr):
 
     return codestr
 
+
+# Basically pickle, in python 2.7.3 at least, does badly with data duplication 
+# upon pickling and unpickling. Combine this with duplicate objects and things
+# are a mess.
+#
+# When the sets are originally created, python calls intern() on the set keys
+# which significantly improves memory usage. Sadly the pickle/unpickle process
+# doesn't call intern() on the keys and results in the same strings being duplicated
+# in memory. This also means pickle will save the same string multiple times in
+# the cache file.
+#
+# By having shell and python cacheline objects with setstate/getstate, we force
+# the object creation through our own routine where we can call intern (via internSet).
+#
+# We also use hashable frozensets and ensure we use references to these so that
+# duplicates can be removed, both in memory and in the resulting pickled data.
+#
+# By playing these games, the size of the cache file shrinks dramatically
+# meaning faster load times and the reloaded cache files also consume much less
+# memory. Smaller cache files, faster load times and lower memory usage is good.
+#
 # A custom getstate/setstate using tuples is actually worth 15% cachesize by
 # avoiding duplication of the attribute names!
 
-
 class SetCache(object):
     def __init__(self):
         self.setcache = {}
@@ -122,7 +140,7 @@ class CodeParserCache(MultiProcessCache):
     # so that an existing cache gets invalidated. Additionally you'll need
     # to increment __cache_version__ in cache.py in order to ensure that old
     # recipe caches don't trigger "Taskhash mismatch" errors.
-    CACHE_VERSION = 11
+    CACHE_VERSION = 9
 
     def __init__(self):
         MultiProcessCache.__init__(self)
@@ -196,7 +214,7 @@ class BufferedLogger(Logger):
         self.buffer = []
 
 class PythonParser():
-    getvars = (".getVar", ".appendVar", ".prependVar", "oe.utils.conditional")
+    getvars = (".getVar", ".appendVar", ".prependVar")
     getvarflags = (".getVarFlag", ".appendVarFlag", ".prependVarFlag")
     containsfuncs = ("bb.utils.contains", "base_contains")
     containsanyfuncs = ("bb.utils.contains_any",  "bb.utils.filter")
@@ -350,9 +368,8 @@ class ShellParser():
     def _parse_shell(self, value):
         try:
             tokens, _ = pyshyacc.parse(value, eof=True, debug=False)
-        except Exception:
-            bb.error('Error during parse shell code, the last 5 lines are:\n%s' % '\n'.join(value.split('\n')[-5:]))
-            raise
+        except pyshlex.NeedMore:
+            raise sherrors.ShellSyntaxError("Unexpected EOF")
 
         self.process_tokens(tokens)
 

bitbake/lib/bb/command.py

@@ -6,8 +6,18 @@ Provide an interface to interact with the bitbake server through 'commands'
 
 # Copyright (C) 2006-2007  Richard Purdie
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 """
 The bitbake server takes 'commands' from its UI/commandline.
@@ -20,7 +30,6 @@ Commands are queued in a CommandQueue
 
 from collections import OrderedDict, defaultdict
 
-import io
 import bb.event
 import bb.cooker
 import bb.remotedata
@@ -66,7 +75,7 @@ class Command:
             # Can run synchronous commands straight away
             command_method = getattr(self.cmds_sync, command)
             if ro_only:
-                if not hasattr(command_method, 'readonly') or not getattr(command_method, 'readonly'):
+                if not hasattr(command_method, 'readonly') or False == getattr(command_method, 'readonly'):
                     return None, "Not able to execute not readonly commands in readonly mode"
             try:
                 self.cooker.process_inotify_updates()
@@ -75,12 +84,8 @@ class Command:
                 result = command_method(self, commandline)
             except CommandError as exc:
                 return None, exc.args[0]
-            except (Exception, SystemExit) as exc:
+            except (Exception, SystemExit):
                 import traceback
-                if isinstance(exc, bb.BBHandledException):
-                    # We need to start returning real exceptions here. Until we do, we can't
-                    # tell if an exception is an instance of bb.BBHandledException
-                    return None, "bb.BBHandledException()\n" + traceback.format_exc()
                 return None, traceback.format_exc()
             else:
                 return result, None
@@ -419,11 +424,7 @@ class CommandsSync:
     getAllAppends.readonly = True
 
     def findProviders(self, command, params):
-        try:
-            mc = params[0]
-        except IndexError:
-            mc = ''
-        return command.cooker.findProviders(mc)
+        return command.cooker.findProviders()
     findProviders.readonly = True
 
     def findBestProvider(self, command, params):
@@ -455,49 +456,54 @@ class CommandsSync:
         return all_p, best
     getRuntimeProviders.readonly = True
 
-    def dataStoreConnectorCmd(self, command, params):
+    def dataStoreConnectorFindVar(self, command, params):
         dsindex = params[0]
-        method = params[1]
-        args = params[2]
-        kwargs = params[3]
+        name = params[1]
+        datastore = command.remotedatastores[dsindex]
+        value, overridedata = datastore._findVar(name)
 
-        d = command.remotedatastores[dsindex]
-        ret = getattr(d, method)(*args, **kwargs)
+        if value:
+            content = value.get('_content', None)
+            if isinstance(content, bb.data_smart.DataSmart):
+                # Value is a datastore (e.g. BB_ORIGENV) - need to handle this carefully
+                idx = command.remotedatastores.check_store(content, True)
+                return {'_content': DataStoreConnectionHandle(idx),
+                        '_connector_origtype': 'DataStoreConnectionHandle',
+                        '_connector_overrides': overridedata}
+            elif isinstance(content, set):
+                return {'_content': list(content),
+                        '_connector_origtype': 'set',
+                        '_connector_overrides': overridedata}
+            else:
+                value['_connector_overrides'] = overridedata
+        else:
+            value = {}
+            value['_connector_overrides'] = overridedata
+        return value
+    dataStoreConnectorFindVar.readonly = True
 
-        if isinstance(ret, bb.data_smart.DataSmart):
-            idx = command.remotedatastores.store(ret)
-            return DataStoreConnectionHandle(idx)
-
-        return ret
-
-    def dataStoreConnectorVarHistCmd(self, command, params):
+    def dataStoreConnectorGetKeys(self, command, params):
         dsindex = params[0]
-        method = params[1]
-        args = params[2]
-        kwargs = params[3]
+        datastore = command.remotedatastores[dsindex]
+        return list(datastore.keys())
+    dataStoreConnectorGetKeys.readonly = True
 
-        d = command.remotedatastores[dsindex].varhistory
-        return getattr(d, method)(*args, **kwargs)
-
-    def dataStoreConnectorVarHistCmdEmit(self, command, params):
+    def dataStoreConnectorGetVarHistory(self, command, params):
         dsindex = params[0]
-        var = params[1]
-        oval = params[2]
-        val = params[3]
-        d = command.remotedatastores[params[4]]
+        name = params[1]
+        datastore = command.remotedatastores[dsindex]
+        return datastore.varhistory.variable(name)
+    dataStoreConnectorGetVarHistory.readonly = True
 
-        o = io.StringIO()
-        command.remotedatastores[dsindex].varhistory.emit(var, oval, val, o, d)
-        return o.getvalue()
+    def dataStoreConnectorExpandPythonRef(self, command, params):
+        config_data_dict = params[0]
+        varname = params[1]
+        expr = params[2]
 
-    def dataStoreConnectorIncHistCmd(self, command, params):
-        dsindex = params[0]
-        method = params[1]
-        args = params[2]
-        kwargs = params[3]
+        config_data = command.remotedatastores.receive_datastore(config_data_dict)
 
-        d = command.remotedatastores[dsindex].inchistory
-        return getattr(d, method)(*args, **kwargs)
+        varparse = bb.data_smart.VariableParse(varname, config_data)
+        return varparse.python_sub(expr)
 
     def dataStoreConnectorRelease(self, command, params):
         dsindex = params[0]
@@ -505,6 +511,31 @@ class CommandsSync:
             raise CommandError('dataStoreConnectorRelease: invalid index %d' % dsindex)
         command.remotedatastores.release(dsindex)
 
+    def dataStoreConnectorSetVarFlag(self, command, params):
+        dsindex = params[0]
+        name = params[1]
+        flag = params[2]
+        value = params[3]
+        datastore = command.remotedatastores[dsindex]
+        datastore.setVarFlag(name, flag, value)
+
+    def dataStoreConnectorDelVar(self, command, params):
+        dsindex = params[0]
+        name = params[1]
+        datastore = command.remotedatastores[dsindex]
+        if len(params) > 2:
+            flag = params[2]
+            datastore.delVarFlag(name, flag)
+        else:
+            datastore.delVar(name)
+
+    def dataStoreConnectorRenameVar(self, command, params):
+        dsindex = params[0]
+        name = params[1]
+        newname = params[2]
+        datastore = command.remotedatastores[dsindex]
+        datastore.renameVar(name, newname)
+
     def parseRecipeFile(self, command, params):
         """
         Parse the specified recipe file (with or without bbappends)
@@ -515,7 +546,8 @@ class CommandsSync:
         appends = params[1]
         appendlist = params[2]
         if len(params) > 3:
-            config_data = command.remotedatastores[params[3]]
+            config_data_dict = params[3]
+            config_data = command.remotedatastores.receive_datastore(config_data_dict)
         else:
             config_data = None
 
@@ -636,16 +668,6 @@ class CommandsAsync:
         command.finishAsyncCommand()
     findFilesMatchingInDir.needcache = False
 
-    def testCookerCommandEvent(self, command, params):
-        """
-        Dummy command used by OEQA selftest to test tinfoil without IO
-        """
-        pattern = params[0]
-
-        command.cooker.testCookerCommandEvent(pattern)
-        command.finishAsyncCommand()
-    testCookerCommandEvent.needcache = False
-
     def findConfigFilePath(self, command, params):
         """
         Find the path of the requested configuration file

bitbake/lib/bb/compat.py

@@ -0,0 +1,6 @@
+"""Code pulled from future python versions, here for compatibility"""
+
+from collections import MutableMapping, KeysView, ValuesView, ItemsView, OrderedDict
+from functools import total_ordering
+
+

bitbake/lib/bb/cooker.py

@@ -1,3 +1,6 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
 # Copyright (C) 2003, 2004  Chris Larson
 # Copyright (C) 2003, 2004  Phil Blundell
@@ -6,27 +9,42 @@
 # Copyright (C) 2005        ROAD GmbH
 # Copyright (C) 2006 - 2007 Richard Purdie
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
 
 import sys, os, glob, os.path, re, time
+import atexit
 import itertools
 import logging
 import multiprocessing
+import sre_constants
 import threading
 from io import StringIO, UnsupportedOperation
 from contextlib import closing
+from functools import wraps
 from collections import defaultdict, namedtuple
 import bb, bb.exceptions, bb.command
 from bb import utils, data, parse, event, cache, providers, taskdata, runqueue, build
 import queue
 import signal
+import subprocess
+import errno
 import prserv.serv
 import pyinotify
 import json
 import pickle
 import codecs
-import hashserv
 
 logger      = logging.getLogger("BitBake")
 collectlog  = logging.getLogger("BitBake.Collection")
@@ -157,45 +175,27 @@ class BBCooker:
 
         self.configuration = configuration
 
-        bb.debug(1, "BBCooker starting %s" % time.time())
-        sys.stdout.flush()
-
         self.configwatcher = pyinotify.WatchManager()
-        bb.debug(1, "BBCooker pyinotify1 %s" % time.time())
-        sys.stdout.flush()
-
-        self.configwatcher.bbseen = set()
-        self.configwatcher.bbwatchedfiles = set()
+        self.configwatcher.bbseen = []
+        self.configwatcher.bbwatchedfiles = []
         self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
-        bb.debug(1, "BBCooker pyinotify2 %s" % time.time())
-        sys.stdout.flush()
         self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
                          pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \
                          pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO
         self.watcher = pyinotify.WatchManager()
-        bb.debug(1, "BBCooker pyinotify3 %s" % time.time())
-        sys.stdout.flush()
-        self.watcher.bbseen = set()
-        self.watcher.bbwatchedfiles = set()
+        self.watcher.bbseen = []
+        self.watcher.bbwatchedfiles = []
         self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
 
-        bb.debug(1, "BBCooker pyinotify complete %s" % time.time())
-        sys.stdout.flush()
-
         # If being called by something like tinfoil, we need to clean cached data
         # which may now be invalid
         bb.parse.clear_cache()
         bb.parse.BBHandler.cached_statements = {}
 
         self.ui_cmdline = None
-        self.hashserv = None
-        self.hashservaddr = None
 
         self.initConfigurationData()
 
-        bb.debug(1, "BBCooker parsed base configuration %s" % time.time())
-        sys.stdout.flush()
-
         # we log all events to a file if so directed
         if self.configuration.writeeventlog:
             # register the log file writer as UI Handler
@@ -233,9 +233,6 @@ class BBCooker:
         # Let SIGHUP exit as SIGTERM
         signal.signal(signal.SIGHUP, self.sigterm_exception)
 
-        bb.debug(1, "BBCooker startup complete %s" % time.time())
-        sys.stdout.flush()
-
     def process_inotify_updates(self):
         for n in [self.confignotifier, self.notifier]:
             if n.check_events(timeout=0):
@@ -273,14 +270,14 @@ class BBCooker:
         if not watcher:
             watcher = self.watcher
         for i in deps:
-            watcher.bbwatchedfiles.add(i[0])
+            watcher.bbwatchedfiles.append(i[0])
             if dirs:
                 f = i[0]
             else:
                 f = os.path.dirname(i[0])
             if f in watcher.bbseen:
                 continue
-            watcher.bbseen.add(f)
+            watcher.bbseen.append(f)
             watchtarget = None
             while True:
                 # We try and add watches for files that don't exist but if they did, would influence
@@ -289,7 +286,7 @@ class BBCooker:
                 try:
                     watcher.add_watch(f, self.watchmask, quiet=False)
                     if watchtarget:
-                        watcher.bbwatchedfiles.add(watchtarget)
+                        watcher.bbwatchedfiles.append(watchtarget)
                     break
                 except pyinotify.WatchManagerError as e:
                     if 'ENOENT' in str(e):
@@ -297,7 +294,7 @@ class BBCooker:
                         f = os.path.dirname(f)
                         if f in watcher.bbseen:
                             break
-                        watcher.bbseen.add(f)
+                        watcher.bbseen.append(f)
                         continue
                     if 'ENOSPC' in str(e):
                         providerlog.error("No space left on device or exceeds fs.inotify.max_user_watches?")
@@ -366,12 +363,17 @@ class BBCooker:
 
         self.data.setVar('BB_CMDLINE', self.ui_cmdline)
 
+        #
+        # Copy of the data store which has been expanded.
+        # Used for firing events and accessing variables where expansion needs to be accounted for
+        #
+        bb.parse.init_parser(self.data)
+
         if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
             self.disableDataTracking()
 
-        for mc in self.databuilder.mcdata.values():
-            mc.renameVar("__depends", "__base_depends")
-            self.add_filewatch(mc.getVar("__base_depends", False), self.configwatcher)
+        self.data.renameVar("__depends", "__base_depends")
+        self.add_filewatch(self.data.getVar("__base_depends", False), self.configwatcher)
 
         self.baseconfig_valid = True
         self.parsecache_valid = False
@@ -383,22 +385,6 @@ class BBCooker:
         except prserv.serv.PRServiceConfigError as e:
             bb.fatal("Unable to start PR Server, exitting")
 
-        if self.data.getVar("BB_HASHSERVE") == "auto":
-            # Create a new hash server bound to a unix domain socket
-            if not self.hashserv:
-                dbfile = (self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")) + "/hashserv.db"
-                self.hashservaddr = "unix://%s/hashserve.sock" % self.data.getVar("TOPDIR")
-                self.hashserv = hashserv.create_server(self.hashservaddr, dbfile, sync=False)
-                self.hashserv.process = multiprocessing.Process(target=self.hashserv.serve_forever)
-                self.hashserv.process.start()
-            self.data.setVar("BB_HASHSERVE", self.hashservaddr)
-            self.databuilder.origdata.setVar("BB_HASHSERVE", self.hashservaddr)
-            self.databuilder.data.setVar("BB_HASHSERVE", self.hashservaddr)
-            for mc in self.databuilder.mcdata:
-                self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", self.hashservaddr)
-
-        bb.parse.init_parser(self.data)
-
     def enableDataTracking(self):
         self.configuration.tracking = True
         if hasattr(self, "data"):
@@ -410,7 +396,10 @@ class BBCooker:
             self.data.disableTracking()
 
     def parseConfiguration(self):
-        self.updateCacheSync()
+        # Set log file verbosity
+        verboselogs = bb.utils.to_boolean(self.data.getVar("BB_VERBOSE_LOGS", False))
+        if verboselogs:
+            bb.msg.loggerVerboseLogs = True
 
         # Change nice level if we're asked to
         nice = self.data.getVar("BB_NICE_LEVEL")
@@ -499,7 +488,6 @@ class BBCooker:
         """
         fn = None
         envdata = None
-        mc = ''
         if not pkgs_to_build:
             pkgs_to_build = []
 
@@ -508,12 +496,6 @@ class BBCooker:
             self.enableDataTracking()
             self.reset()
 
-        def mc_base(p):
-            if p.startswith('mc:'):
-                s = p.split(':')
-                if len(s) == 2:
-                    return s[1]
-            return None
 
         if buildfile:
             # Parse the configuration here. We need to do it explicitly here since
@@ -524,16 +506,16 @@ class BBCooker:
             fn = self.matchFile(fn)
             fn = bb.cache.realfn2virtual(fn, cls, mc)
         elif len(pkgs_to_build) == 1:
-            mc = mc_base(pkgs_to_build[0])
-            if not mc:
-                ignore = self.data.getVar("ASSUME_PROVIDED") or ""
-                if pkgs_to_build[0] in set(ignore.split()):
-                    bb.fatal("%s is in ASSUME_PROVIDED" % pkgs_to_build[0])
+            ignore = self.data.getVar("ASSUME_PROVIDED") or ""
+            if pkgs_to_build[0] in set(ignore.split()):
+                bb.fatal("%s is in ASSUME_PROVIDED" % pkgs_to_build[0])
 
-                taskdata, runlist = self.buildTaskData(pkgs_to_build, None, self.configuration.abort, allowincomplete=True)
+            taskdata, runlist = self.buildTaskData(pkgs_to_build, None, self.configuration.abort, allowincomplete=True)
 
-                mc = runlist[0][0]
-                fn = runlist[0][3]
+            mc = runlist[0][0]
+            fn = runlist[0][3]
+        else:
+            envdata = self.data
 
         if fn:
             try:
@@ -542,12 +524,6 @@ class BBCooker:
             except Exception as e:
                 parselog.exception("Unable to read %s", fn)
                 raise
-        else:
-            if not mc in self.databuilder.mcdata:
-                bb.fatal('Not multiconfig named "%s" found' % mc)
-            envdata = self.databuilder.mcdata[mc]
-            data.expandKeys(envdata)
-            parse.ast.runAnonFuncs(envdata)
 
         # Display history
         with closing(StringIO()) as env:
@@ -560,6 +536,7 @@ class BBCooker:
             logger.plain(env.getvalue())
 
         # emit the metadata which isnt valid shell
+        data.expandKeys(envdata)
         for e in sorted(envdata.keys()):
             if envdata.getVarFlag(e, 'func', False) and envdata.getVarFlag(e, 'python', False):
                 logger.plain("\npython %s () {\n%s}\n", e, envdata.getVar(e, False))
@@ -587,10 +564,10 @@ class BBCooker:
         wildcard = False
 
         # Wild card expansion:
-        # Replace string such as "mc:*:bash"
-        # into "mc:A:bash mc:B:bash bash"
+        # Replace string such as "multiconfig:*:bash"
+        # into "multiconfig:A:bash multiconfig:B:bash bash"
         for k in targetlist:
-            if k.startswith("mc:"):
+            if k.startswith("multiconfig:"):
                 if wildcard:
                     bb.fatal('multiconfig conflict')
                 if k.split(":")[1] == "*":
@@ -623,7 +600,7 @@ class BBCooker:
         runlist = []
         for k in fulltargetlist:
             mc = ""
-            if k.startswith("mc:"):
+            if k.startswith("multiconfig:"):
                 mc = k.split(":")[1]
                 k = ":".join(k.split(":")[2:])
             ktask = task
@@ -642,37 +619,6 @@ class BBCooker:
             runlist.append([mc, k, ktask, fn])
             bb.event.fire(bb.event.TreeDataPreparationProgress(current, len(fulltargetlist)), self.data)
 
-        havemc = False
-        for mc in self.multiconfigs:
-            if taskdata[mc].get_mcdepends():
-                havemc = True
-
-        # No need to do check providers if there are no mcdeps or not an mc build
-        if havemc or len(self.multiconfigs) > 1:
-            seen = set()
-            new = True
-            # Make sure we can provide the multiconfig dependency
-            while new:
-                mcdeps = set()
-                # Add unresolved first, so we can get multiconfig indirect dependencies on time
-                for mc in self.multiconfigs:
-                    taskdata[mc].add_unresolved(localdata[mc], self.recipecaches[mc])
-                    mcdeps |= set(taskdata[mc].get_mcdepends())
-                new = False
-                for mc in self.multiconfigs:
-                    for k in mcdeps:
-                        if k in seen:
-                            continue
-                        l = k.split(':')
-                        depmc = l[2]
-                        if depmc not in self.multiconfigs:
-                            bb.fatal("Multiconfig dependency %s depends on nonexistent mc configuration %s" % (k,depmc))
-                        else:
-                            logger.debug(1, "Adding providers for multiconfig dependency %s" % l[3])
-                            taskdata[depmc].add_provider(localdata[depmc], self.recipecaches[depmc], l[3])
-                            seen.add(k)
-                            new = True
-
         for mc in self.multiconfigs:
             taskdata[mc].add_unresolved(localdata[mc], self.recipecaches[mc])
 
@@ -708,7 +654,7 @@ class BBCooker:
     @staticmethod
     def add_mc_prefix(mc, pn):
         if mc:
-            return "mc:%s:%s" % (mc, pn)
+            return "multiconfig:%s:%s" % (mc, pn)
         return pn
 
     def buildDependTree(self, rq, taskdata):
@@ -759,8 +705,8 @@ class BBCooker:
             if not dotname in depend_tree["tdepends"]:
                 depend_tree["tdepends"][dotname] = []
             for dep in rq.rqdata.runtaskentries[tid].depends:
-                (depmc, depfn, _, deptaskfn) = bb.runqueue.split_tid_mcfn(dep)
-                deppn = self.recipecaches[depmc].pkg_fn[deptaskfn]
+                (depmc, depfn, deptaskname, deptaskfn) = bb.runqueue.split_tid_mcfn(dep)
+                deppn = self.recipecaches[mc].pkg_fn[deptaskfn]
                 depend_tree["tdepends"][dotname].append("%s.%s" % (deppn, bb.runqueue.taskname_from_tid(dep)))
             if taskfn not in seen_fns:
                 seen_fns.append(taskfn)
@@ -907,23 +853,40 @@ class BBCooker:
             os.unlink('package-depends.dot')
         except FileNotFoundError:
             pass
-        try:
-            os.unlink('recipe-depends.dot')
-        except FileNotFoundError:
-            pass
 
         with open('task-depends.dot', 'w') as f:
             f.write("digraph depends {\n")
-            for task in sorted(depgraph["tdepends"]):
+            for task in depgraph["tdepends"]:
                 (pn, taskname) = task.rsplit(".", 1)
                 fn = depgraph["pn"][pn]["filename"]
                 version = depgraph["pn"][pn]["version"]
                 f.write('"%s.%s" [label="%s %s\\n%s\\n%s"]\n' % (pn, taskname, pn, taskname, version, fn))
-                for dep in sorted(depgraph["tdepends"][task]):
+                for dep in depgraph["tdepends"][task]:
                     f.write('"%s" -> "%s"\n' % (task, dep))
             f.write("}\n")
         logger.info("Task dependencies saved to 'task-depends.dot'")
 
+        with open('recipe-depends.dot', 'w') as f:
+            f.write("digraph depends {\n")
+            pndeps = {}
+            for task in depgraph["tdepends"]:
+                (pn, taskname) = task.rsplit(".", 1)
+                if pn not in pndeps:
+                    pndeps[pn] = set()
+                for dep in depgraph["tdepends"][task]:
+                    (deppn, deptaskname) = dep.rsplit(".", 1)
+                    pndeps[pn].add(deppn)
+            for pn in pndeps:
+                fn = depgraph["pn"][pn]["filename"]
+                version = depgraph["pn"][pn]["version"]
+                f.write('"%s" [label="%s\\n%s\\n%s"]\n' % (pn, pn, version, fn))
+                for dep in pndeps[pn]:
+                    if dep == pn:
+                        continue
+                    f.write('"%s" -> "%s"\n' % (pn, dep))
+            f.write("}\n")
+        logger.info("Flatened recipe dependencies saved to 'recipe-depends.dot'")
+
     def show_appends_with_no_recipes(self):
         # Determine which bbappends haven't been applied
 
@@ -1018,22 +981,17 @@ class BBCooker:
         if matches:
             bb.event.fire(bb.event.FilesMatchingFound(filepattern, matches), self.data)
 
-    def testCookerCommandEvent(self, filepattern):
-        # Dummy command used by OEQA selftest to test tinfoil without IO
-        matches = ["A", "B"]
-        bb.event.fire(bb.event.FilesMatchingFound(filepattern, matches), self.data)
-
     def findProviders(self, mc=''):
-        return bb.providers.findProviders(self.databuilder.mcdata[mc], self.recipecaches[mc], self.recipecaches[mc].pkg_pn)
+        return bb.providers.findProviders(self.data, self.recipecaches[mc], self.recipecaches[mc].pkg_pn)
 
     def findBestProvider(self, pn, mc=''):
         if pn in self.recipecaches[mc].providers:
             filenames = self.recipecaches[mc].providers[pn]
-            eligible, foundUnique = bb.providers.filterProviders(filenames, pn, self.databuilder.mcdata[mc], self.recipecaches[mc])
+            eligible, foundUnique = bb.providers.filterProviders(filenames, pn, self.data, self.recipecaches[mc])
             filename = eligible[0]
             return None, None, None, filename
         elif pn in self.recipecaches[mc].pkg_pn:
-            return bb.providers.findBestProvider(pn, self.databuilder.mcdata[mc], self.recipecaches[mc], self.recipecaches[mc].pkg_pn)
+            return bb.providers.findBestProvider(pn, self.data, self.recipecaches[mc], self.recipecaches[mc].pkg_pn)
         else:
             return None, None, None, None
 
@@ -1205,13 +1163,12 @@ class BBCooker:
             for c in collection_list:
                 calc_layer_priority(c)
                 regex = self.data.getVar("BBFILE_PATTERN_%s" % c)
-                if regex is None:
+                if regex == None:
                     parselog.error("BBFILE_PATTERN_%s not defined" % c)
                     errors = True
                     continue
                 elif regex == "":
                     parselog.debug(1, "BBFILE_PATTERN_%s is empty" % c)
-                    cre = re.compile('^NULL$')
                     errors = False
                 else:
                     try:
@@ -1311,7 +1268,7 @@ class BBCooker:
         self.parseConfiguration()
 
         # If we are told to do the None task then query the default task
-        if task is None:
+        if (task == None):
             task = self.configuration.cmd
         if not task.startswith("do_"):
             task = "do_%s" % task
@@ -1455,7 +1412,7 @@ class BBCooker:
         self.buildSetVars()
 
         # If we are told to do the None task then query the default task
-        if task is None:
+        if (task == None):
             task = self.configuration.cmd
 
         if not task.startswith("do_"):
@@ -1473,7 +1430,7 @@ class BBCooker:
         ntargets = []
         for target in runlist:
             if target[0]:
-                ntargets.append("mc:%s:%s:%s" % (target[0], target[1], target[2]))
+                ntargets.append("multiconfig:%s:%s:%s" % (target[0], target[1], target[2]))
             ntargets.append("%s:%s" % (target[1], target[2]))
 
         for mc in self.multiconfigs:
@@ -1593,12 +1550,9 @@ class BBCooker:
             raise NothingToBuild
 
         ignore = (self.data.getVar("ASSUME_PROVIDED") or "").split()
-        for pkg in pkgs_to_build.copy():
+        for pkg in pkgs_to_build:
             if pkg in ignore:
                 parselog.warning("Explicit target \"%s\" is in ASSUME_PROVIDED, ignoring" % pkg)
-            if pkg.startswith("multiconfig:"):
-                pkgs_to_build.remove(pkg)
-                pkgs_to_build.append(pkg.replace("multiconfig:", "mc:"))
 
         if 'world' in pkgs_to_build:
             pkgs_to_build.remove('world')
@@ -1606,11 +1560,11 @@ class BBCooker:
                 bb.providers.buildWorldTargetList(self.recipecaches[mc], task)
                 for t in self.recipecaches[mc].world_target:
                     if mc:
-                        t = "mc:" + mc + ":" + t
+                        t = "multiconfig:" + mc + ":" + t
                     pkgs_to_build.append(t)
 
         if 'universe' in pkgs_to_build:
-            parselog.verbnote("The \"universe\" target is only intended for testing and may produce errors.")
+            parselog.warning("The \"universe\" target is only intended for testing and may produce errors.")
             parselog.debug(1, "collating packages for \"universe\"")
             pkgs_to_build.remove('universe')
             for mc in self.multiconfigs:
@@ -1625,7 +1579,7 @@ class BBCooker:
                             bb.debug(1, "Skipping %s for universe tasks as task %s doesn't exist" % (t, task))
                             continue
                     if mc:
-                        t = "mc:" + mc + ":" + t
+                        t = "multiconfig:" + mc + ":" + t
                     pkgs_to_build.append(t)
 
         return pkgs_to_build
@@ -1637,13 +1591,10 @@ class BBCooker:
         return
 
     def post_serve(self):
-        self.shutdown(force=True)
         prserv.serv.auto_shutdown()
-        if self.hashserv:
-            self.hashserv.process.terminate()
-            self.hashserv.process.join()
         bb.event.fire(CookerExit(), self.data)
 
+
     def shutdown(self, force = False):
         if force:
             self.state = state.forceshutdown
@@ -1652,14 +1603,14 @@ class BBCooker:
 
         if self.parser:
             self.parser.shutdown(clean=not force, force=force)
-            self.parser.final_cleanup()
+        self.notifier.stop()
+        self.confignotifier.stop()
 
     def finishcommand(self):
         self.state = state.initial
 
     def reset(self):
         self.initConfigurationData()
-        self.handlePRServ()
 
     def clientComplete(self):
         """Called when the client is done using the server"""
@@ -1668,8 +1619,6 @@ class BBCooker:
         self.command.reset()
         self.databuilder.reset()
         self.data = self.databuilder.data
-        self.parsecache_valid = False
-        self.baseconfig_valid = False
 
 
 class CookerExit(bb.event.Event):
@@ -1684,15 +1633,12 @@ class CookerExit(bb.event.Event):
 class CookerCollectFiles(object):
     def __init__(self, priorities):
         self.bbappends = []
-        # Priorities is a list of tupples, with the second element as the pattern.
-        # We need to sort the list with the longest pattern first, and so on to
-        # the shortest.  This allows nested layers to be properly evaluated.
-        self.bbfile_config_priorities = sorted(priorities, key=lambda tup: tup[1], reverse=True)
+        self.bbfile_config_priorities = priorities
 
     def calc_bbfile_priority( self, filename, matched = None ):
         for _, _, regex, pri in self.bbfile_config_priorities:
             if regex.match(filename):
-                if matched is not None:
+                if matched != None:
                     if not regex in matched:
                         matched.add(regex)
                 return pri
@@ -1789,12 +1735,12 @@ class CookerCollectFiles(object):
                 # When constructing an older style single regex, it's possible for BBMASK
                 # to end up beginning with '|', which matches and masks _everything_.
                 if mask.startswith("|"):
-                    collectlog.warning("BBMASK contains regular expression beginning with '|', fixing: %s" % mask)
+                    collectlog.warn("BBMASK contains regular expression beginning with '|', fixing: %s" % mask)
                     mask = mask[1:]
                 try:
                     re.compile(mask)
                     bbmasks.append(mask)
-                except re.error:
+                except sre_constants.error:
                     collectlog.critical("BBMASK contains an invalid regular expression, ignoring: %s" % mask)
 
             # Then validate the combined regular expressions. This should never
@@ -1802,7 +1748,7 @@ class CookerCollectFiles(object):
             bbmask = "|".join(bbmasks)
             try:
                 bbmask_compiled = re.compile(bbmask)
-            except re.error:
+            except sre_constants.error:
                 collectlog.critical("BBMASK is not a valid regular expression, ignoring: %s" % bbmask)
                 bbmask = None
 
@@ -1861,25 +1807,21 @@ class CookerCollectFiles(object):
             realfn, cls, mc = bb.cache.virtualfn2realfn(p)
             priorities[p] = self.calc_bbfile_priority(realfn, matched)
 
+        # Don't show the warning if the BBFILE_PATTERN did match .bbappend files
         unmatched = set()
         for _, _, regex, pri in self.bbfile_config_priorities:
             if not regex in matched:
                 unmatched.add(regex)
 
-        # Don't show the warning if the BBFILE_PATTERN did match .bbappend files
-        def find_bbappend_match(regex):
+        def findmatch(regex):
             for b in self.bbappends:
                 (bbfile, append) = b
                 if regex.match(append):
-                    # If the bbappend is matched by already "matched set", return False
-                    for matched_regex in matched:
-                        if matched_regex.match(append):
-                            return False
                     return True
             return False
 
         for unmatch in unmatched.copy():
-            if find_bbappend_match(unmatch):
+            if findmatch(unmatch):
                 unmatched.remove(unmatch)
 
         for collection, pattern, regex, _ in self.bbfile_config_priorities:
@@ -1895,6 +1837,35 @@ class ParsingFailure(Exception):
         self.recipe = recipe
         Exception.__init__(self, realexception, recipe)
 
+class Feeder(multiprocessing.Process):
+    def __init__(self, jobs, to_parsers, quit):
+        self.quit = quit
+        self.jobs = jobs
+        self.to_parsers = to_parsers
+        multiprocessing.Process.__init__(self)
+
+    def run(self):
+        while True:
+            try:
+                quit = self.quit.get_nowait()
+            except queue.Empty:
+                pass
+            else:
+                if quit == 'cancel':
+                    self.to_parsers.cancel_join_thread()
+                break
+
+            try:
+                job = self.jobs.pop()
+            except IndexError:
+                break
+
+            try:
+                self.to_parsers.put(job, timeout=0.5)
+            except queue.Full:
+                self.jobs.insert(0, job)
+                continue
+
 class Parser(multiprocessing.Process):
     def __init__(self, jobs, results, quit, init, profile):
         self.jobs = jobs
@@ -1934,22 +1905,21 @@ class Parser(multiprocessing.Process):
             except queue.Empty:
                 pass
             else:
-                self.results.close()
-                self.results.join_thread()
+                self.results.cancel_join_thread()
                 break
 
             if pending:
                 result = pending.pop()
             else:
                 try:
-                    job = self.jobs.pop()
-                except IndexError:
-                    self.results.close()
-                    self.results.join_thread()
+                    job = self.jobs.get(timeout=0.25)
+                except queue.Empty:
+                    continue
+
+                if job is None:
                     break
                 result = self.parse(*job)
-                # Clear the siggen cache after parsing to control memory usage, its huge
-                bb.parse.siggen.postparsing_clean_cache()
+
             try:
                 self.results.put(result, timeout=0.25)
             except queue.Full:
@@ -1957,7 +1927,6 @@ class Parser(multiprocessing.Process):
 
     def parse(self, filename, appends):
         try:
-            origfilter = bb.event.LogHandler.filter
             # Record the filename we're parsing into any events generated
             def parse_filter(self, record):
                 record.taskpid = bb.event.worker_pid
@@ -1980,8 +1949,6 @@ class Parser(multiprocessing.Process):
         # a SystemExit event for example.
         except BaseException as exc:
             return True, ParsingFailure(exc, filename)
-        finally:
-            bb.event.LogHandler.filter = origfilter
 
 class CookerParser(object):
     def __init__(self, cooker, filelist, masked):
@@ -2021,7 +1988,6 @@ class CookerParser(object):
 
         self.start()
         self.haveshutdown = False
-        self.syncthread = None
 
     def start(self):
         self.results = self.load_cached()
@@ -2034,15 +2000,14 @@ class CookerParser(object):
                 multiprocessing.util.Finalize(None, bb.codeparser.parser_cache_save, exitpriority=1)
                 multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save, exitpriority=1)
 
+            self.feeder_quit = multiprocessing.Queue(maxsize=1)
             self.parser_quit = multiprocessing.Queue(maxsize=self.num_processes)
+            self.jobs = multiprocessing.Queue(maxsize=self.num_processes)
             self.result_queue = multiprocessing.Queue()
-
-            def chunkify(lst,n):
-                return [lst[i::n] for i in range(n)]
-            self.jobs = chunkify(self.willparse, self.num_processes)
-
+            self.feeder = Feeder(self.willparse, self.jobs, self.feeder_quit)
+            self.feeder.start()
             for i in range(0, self.num_processes):
-                parser = Parser(self.jobs[i], self.result_queue, self.parser_quit, init, self.cooker.configuration.profile)
+                parser = Parser(self.jobs, self.result_queue, self.parser_quit, init, self.cooker.configuration.profile)
                 parser.start()
                 self.process_names.append(parser.name)
                 self.processes.append(parser)
@@ -2063,17 +2028,17 @@ class CookerParser(object):
                                             self.total)
 
             bb.event.fire(event, self.cfgdata)
+            self.feeder_quit.put(None)
+            for process in self.processes:
+                self.parser_quit.put(None)
+        else:
+            self.feeder_quit.put('cancel')
 
-        for process in self.processes:
-            self.parser_quit.put(None)
+            self.parser_quit.cancel_join_thread()
+            for process in self.processes:
+                self.parser_quit.put(None)
 
-        # Cleanup the queue before call process.join(), otherwise there might be
-        # deadlocks.
-        while True:
-            try:
-               self.result_queue.get(timeout=0.25)
-            except queue.Empty:
-                break
+            self.jobs.cancel_join_thread()
 
         for process in self.processes:
             if force:
@@ -2081,14 +2046,11 @@ class CookerParser(object):
                 process.terminate()
             else:
                 process.join()
-
-        self.parser_quit.close()
-        # Allow data left in the cancel queue to be discarded
-        self.parser_quit.cancel_join_thread()
+        self.feeder.join()
 
         sync = threading.Thread(target=self.bb_cache.sync)
-        self.syncthread = sync
         sync.start()
+        multiprocessing.util.Finalize(None, sync.join, exitpriority=-100)
         bb.codeparser.parser_cache_savemerge()
         bb.fetch.fetcher_parse_done()
         if self.cooker.configuration.profile:
@@ -2102,10 +2064,6 @@ class CookerParser(object):
             bb.utils.process_profilelog(profiles, pout = pout)
             print("Processed parsing statistics saved to %s" % (pout))
 
-    def final_cleanup(self):
-        if self.syncthread:
-            self.syncthread.join()
-
     def load_cached(self):
         for filename, appends in self.fromcache:
             cached, infos = self.bb_cache.load(filename, appends)
@@ -2138,18 +2096,18 @@ class CookerParser(object):
         except bb.BBHandledException as exc:
             self.error += 1
             logger.error('Failed to parse recipe: %s' % exc.recipe)
-            self.shutdown(clean=False, force=True)
+            self.shutdown(clean=False)
             return False
         except ParsingFailure as exc:
             self.error += 1
             logger.error('Unable to parse %s: %s' %
                      (exc.recipe, bb.exceptions.to_string(exc.realexception)))
-            self.shutdown(clean=False, force=True)
+            self.shutdown(clean=False)
             return False
         except bb.parse.ParseError as exc:
             self.error += 1
             logger.error(str(exc))
-            self.shutdown(clean=False, force=True)
+            self.shutdown(clean=False)
             return False
         except bb.data_smart.ExpansionError as exc:
             self.error += 1
@@ -2158,7 +2116,7 @@ class CookerParser(object):
             tb = list(itertools.dropwhile(lambda e: e.filename.startswith(bbdir), exc.traceback))
             logger.error('ExpansionError during parsing %s', value.recipe,
                          exc_info=(etype, value, tb))
-            self.shutdown(clean=False, force=True)
+            self.shutdown(clean=False)
             return False
         except Exception as exc:
             self.error += 1
@@ -2170,7 +2128,7 @@ class CookerParser(object):
                 # Most likely, an exception occurred during raising an exception
                 import traceback
                 logger.error('Exception during parse: %s' % traceback.format_exc())
-            self.shutdown(clean=False, force=True)
+            self.shutdown(clean=False)
             return False
 
         self.current += 1

bitbake/lib/bb/cookerdata.py

@@ -1,3 +1,6 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 #
 # Copyright (C) 2003, 2004  Chris Larson
 # Copyright (C) 2003, 2004  Phil Blundell
@@ -6,14 +9,23 @@
 # Copyright (C) 2005        ROAD GmbH
 # Copyright (C) 2006        Richard Purdie
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import logging
 import os
 import re
 import sys
-import hashlib
 from functools import wraps
 import bb
 from bb import data
@@ -58,14 +70,11 @@ class ConfigParameters(object):
     def updateToServer(self, server, environment):
         options = {}
         for o in ["abort", "force", "invalidate_stamp",
-                  "debug", "dry_run", "dump_signatures",
+                  "verbose", "debug", "dry_run", "dump_signatures",
                   "debug_domains", "extra_assume_provided", "profile",
                   "prefile", "postfile", "server_timeout"]:
             options[o] = getattr(self.options, o)
 
-        options['build_verbose_shell'] = self.options.verbose
-        options['build_verbose_stdout'] = self.options.verbose
-
         ret, error = server.runCommand(["updateConfig", options, environment, sys.argv])
         if error:
             raise Exception("Unable to update the server configuration with local parameters: %s" % error)
@@ -125,11 +134,8 @@ class CookerConfiguration(object):
         self.profile = False
         self.nosetscene = False
         self.setsceneonly = False
-        self.skipsetscene = False
         self.invalidate_stamp = False
         self.dump_signatures = []
-        self.build_verbose_shell = False
-        self.build_verbose_stdout = False
         self.dry_run = False
         self.tracking = False
         self.xmlrpcinterface = []
@@ -137,8 +143,7 @@ class CookerConfiguration(object):
         self.writeeventlog = False
         self.server_only = False
         self.limited_deps = False
-        self.runall = []
-        self.runonly = []
+        self.runall = None
 
         self.env = {}
 
@@ -273,13 +278,12 @@ class CookerDataBuilder(object):
         self.mcdata = {}
 
     def parseBaseConfiguration(self):
-        data_hash = hashlib.sha256()
         try:
+            bb.parse.init_parser(self.basedata)
             self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles)
 
             if self.data.getVar("BB_WORKERCONTEXT", False) is None:
                 bb.fetch.fetcher_init(self.data)
-            bb.parse.init_parser(self.data)
             bb.codeparser.parser_cache_init(self.data)
 
             bb.event.fire(bb.event.ConfigParsed(), self.data)
@@ -297,21 +301,17 @@ class CookerDataBuilder(object):
                 bb.event.fire(bb.event.ConfigParsed(), self.data)
 
             bb.parse.init_parser(self.data)
-            data_hash.update(self.data.get_hash().encode('utf-8'))
+            self.data_hash = self.data.get_hash()
             self.mcdata[''] = self.data
 
             multiconfig = (self.data.getVar("BBMULTICONFIG") or "").split()
             for config in multiconfig:
-                if config[0].isdigit():
-                    bb.fatal("Multiconfig name '%s' is invalid as multiconfigs cannot start with a digit" % config)
                 mcdata = self.parseConfigurationFiles(self.prefiles, self.postfiles, config)
                 bb.event.fire(bb.event.ConfigParsed(), mcdata)
                 self.mcdata[config] = mcdata
-                data_hash.update(mcdata.get_hash().encode('utf-8'))
             if multiconfig:
                 bb.event.fire(bb.event.MultiConfigParsed(self.mcdata), self.data)
 
-            self.data_hash = data_hash.hexdigest()
         except (SyntaxError, bb.BBHandledException):
             raise bb.BBHandledException
         except bb.data_smart.ExpansionError as e:
@@ -353,27 +353,14 @@ class CookerDataBuilder(object):
             data = parse_config_file(layerconf, data)
 
             layers = (data.getVar('BBLAYERS') or "").split()
-            broken_layers = []
-
-            if not layers:
-                bb.fatal("The bblayers.conf file doesn't contain any BBLAYERS definition")
 
             data = bb.data.createCopy(data)
             approved = bb.utils.approved_variables()
-
-            # Check whether present layer directories exist
             for layer in layers:
                 if not os.path.isdir(layer):
-                    broken_layers.append(layer)
-
-            if broken_layers:
-                parselog.critical("The following layer directories do not exist:")
-                for layer in broken_layers:
-                    parselog.critical("   %s", layer)
-                parselog.critical("Please check BBLAYERS in %s" % (layerconf))
-                sys.exit(1)
-
-            for layer in layers:
+                    parselog.critical("Layer directory '%s' does not exist! "
+                                      "Please check BBLAYERS in %s" % (layer, layerconf))
+                    sys.exit(1)
                 parselog.debug(2, "Adding layer %s", layer)
                 if 'HOME' in approved and '~' in layer:
                     layer = os.path.expanduser(layer)
@@ -403,19 +390,11 @@ class CookerDataBuilder(object):
                 bb.fatal("BBFILES_DYNAMIC entries must be of the form <collection name>:<filename pattern>, not:\n    %s" % "\n    ".join(invalid))
 
             layerseries = set((data.getVar("LAYERSERIES_CORENAMES") or "").split())
-            collections_tmp = collections[:]
             for c in collections:
-                collections_tmp.remove(c)
-                if c in collections_tmp:
-                    bb.fatal("Found duplicated BBFILE_COLLECTIONS '%s', check bblayers.conf or layer.conf to fix it." % c)
                 compat = set((data.getVar("LAYERSERIES_COMPAT_%s" % c) or "").split())
-                if compat and not layerseries:
-                    bb.fatal("No core layer found to work with layer '%s'. Missing entry in bblayers.conf?" % c)
                 if compat and not (compat & layerseries):
                     bb.fatal("Layer %s is not compatible with the core layer which only supports these series: %s (layer is compatible with %s)"
                               % (c, " ".join(layerseries), " ".join(compat)))
-                elif not compat and not data.getVar("BB_WORKERCONTEXT"):
-                    bb.warn("Layer %s should set LAYERSERIES_COMPAT_%s in its conf/layer.conf file to list the core layer names it is compatible with." % (c, c))
 
         if not data.getVar("BBPATH"):
             msg = "The BBPATH variable is not set"

bitbake/lib/bb/daemonize.py

@@ -1,7 +1,3 @@
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
-
 """
 Python Daemonizing helper
 
@@ -20,10 +16,6 @@ def createDaemon(function, logfile):
     background as a daemon, returning control to the caller.
     """
 
-    # Ensure stdout/stderror are flushed before forking to avoid duplicate output
-    sys.stdout.flush()
-    sys.stderr.flush()
-
     try:
         # Fork a child process so the parent can exit.  This returns control to
         # the command-line or shell.  It also guarantees that the child will not
@@ -57,8 +49,8 @@ def createDaemon(function, logfile):
             # exit() or _exit()?
             # _exit is like exit(), but it doesn't call any functions registered
             # with atexit (and on_exit) or any registered signal handlers.  It also
-            # closes any open file descriptors, but doesn't flush any buffered output.
-            # Using exit() may cause all any temporary files to be unexpectedly
+            # closes any open file descriptors.  Using exit() may cause all stdio
+            # streams to be flushed twice and any temporary files may be unexpectedly
             # removed.  It's therefore recommended that child branches of a fork()
             # and the parent branch(es) of a daemon use _exit().
             os._exit(0)
@@ -69,19 +61,17 @@ def createDaemon(function, logfile):
     # The second child.
 
     # Replace standard fds with our own
-    with open('/dev/null', 'r') as si:
-        os.dup2(si.fileno(), sys.stdin.fileno())
+    si = open('/dev/null', 'r')
+    os.dup2(si.fileno(), sys.stdin.fileno())
 
     try:
         so = open(logfile, 'a+')
+        se = so
         os.dup2(so.fileno(), sys.stdout.fileno())
-        os.dup2(so.fileno(), sys.stderr.fileno())
+        os.dup2(se.fileno(), sys.stderr.fileno())
     except io.UnsupportedOperation:
         sys.stdout = open(logfile, 'a+')
-
-    # Have stdout and stderr be the same so log output matches chronologically
-    # and there aren't two seperate buffers
-    sys.stderr = sys.stdout
+        sys.stderr = sys.stdout
 
     try:
         function()
@@ -89,9 +79,4 @@ def createDaemon(function, logfile):
         traceback.print_exc()
     finally:
         bb.event.print_ui_queue()
-        # os._exit() doesn't flush open files like os.exit() does. Manually flush
-        # stdout and stderr so that any logging output will be seen, particularly
-        # exception tracebacks.
-        sys.stdout.flush()
-        sys.stderr.flush()
         os._exit(0)

bitbake/lib/bb/data.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 """
 BitBake 'Data' implementations
 
@@ -20,12 +22,22 @@ the speed is more critical here.
 # Copyright (C) 2003, 2004  Chris Larson
 # Copyright (C) 2005        Holger Hans Peter Freyther
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 #
 # Based on functions from the base bb module, Copyright 2003 Holger Schurig
 
 import sys, os, re
-import hashlib
 if sys.argv[0][-5:] == "pydoc":
     path = os.path.dirname(os.path.dirname(sys.argv[1]))
 else:
@@ -79,7 +91,7 @@ def expand(s, d, varname = None):
     return d.expand(s, varname)
 
 def expandKeys(alterdata, readdata = None):
-    if readdata is None:
+    if readdata == None:
         readdata = alterdata
 
     todolist = {}
@@ -130,7 +142,7 @@ def emit_var(var, o=sys.__stdout__, d = init(), all=False):
         if all:
             oval = d.getVar(var, False)
         val = d.getVar(var)
-    except (KeyboardInterrupt):
+    except (KeyboardInterrupt, bb.build.FuncFailed):
         raise
     except Exception as exc:
         o.write('# expansion of %s threw %s: %s\n' % (var, exc.__class__.__name__, str(exc)))
@@ -271,12 +283,14 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d):
     try:
         if key[-1] == ']':
             vf = key[:-1].split('[')
-            value, parser = d.getVarFlag(vf[0], vf[1], False, retparser=True)
+            value = d.getVarFlag(vf[0], vf[1], False)
+            parser = d.expandWithRefs(value, key)
             deps |= parser.references
             deps = deps | (keys & parser.execs)
             return deps, value
         varflags = d.getVarFlags(key, ["vardeps", "vardepvalue", "vardepsexclude", "exports", "postfuncs", "prefuncs", "lineno", "filename"]) or {}
         vardeps = varflags.get("vardeps")
+        value = d.getVarFlag(key, "_content", False)
 
         def handle_contains(value, contains, d):
             newvalue = ""
@@ -295,35 +309,25 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d):
                 return newvalue
             return value + newvalue
 
-        def handle_remove(value, deps, removes, d):
-            for r in sorted(removes):
-                r2 = d.expandWithRefs(r, None)
-                value += "\n_remove of %s" % r
-                deps |= r2.references
-                deps = deps | (keys & r2.execs)
-                value = handle_contains(value, r2.contains, d)
-            return value
-
         if "vardepvalue" in varflags:
-            value = varflags.get("vardepvalue")
+           value = varflags.get("vardepvalue")
         elif varflags.get("func"):
             if varflags.get("python"):
-                value = d.getVarFlag(key, "_content", False)
                 parser = bb.codeparser.PythonParser(key, logger)
+                if value and "\t" in value:
+                    logger.warning("Variable %s contains tabs, please remove these (%s)" % (key, d.getVar("FILE")))
                 parser.parse_python(value, filename=varflags.get("filename"), lineno=varflags.get("lineno"))
                 deps = deps | parser.references
                 deps = deps | (keys & parser.execs)
                 value = handle_contains(value, parser.contains, d)
             else:
-                value, parsedvar = d.getVarFlag(key, "_content", False, retparser=True)
+                parsedvar = d.expandWithRefs(value, key)
                 parser = bb.codeparser.ShellParser(key, logger)
                 parser.parse_shell(parsedvar.value)
                 deps = deps | shelldeps
                 deps = deps | parsedvar.references
                 deps = deps | (keys & parser.execs) | (keys & parsedvar.execs)
                 value = handle_contains(value, parsedvar.contains, d)
-                if hasattr(parsedvar, "removes"):
-                    value = handle_remove(value, deps, parsedvar.removes, d)
             if vardeps is None:
                 parser.log.flush()
             if "prefuncs" in varflags:
@@ -333,12 +337,10 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d):
             if "exports" in varflags:
                 deps = deps | set(varflags["exports"].split())
         else:
-            value, parser = d.getVarFlag(key, "_content", False, retparser=True)
+            parser = d.expandWithRefs(value, key)
             deps |= parser.references
             deps = deps | (keys & parser.execs)
             value = handle_contains(value, parser.contains, d)
-            if hasattr(parser, "removes"):
-                value = handle_remove(value, deps, parser.removes, d)
 
         if "vardepvalueexclude" in varflags:
             exclude = varflags.get("vardepvalueexclude")
@@ -366,7 +368,7 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d):
     #bb.note("Variable %s references %s and calls %s" % (key, str(deps), str(execs)))
     #d.setVarFlag(key, "vardeps", deps)
 
-def generate_dependencies(d, whitelist):
+def generate_dependencies(d):
 
     keys = set(key for key in d if not key.startswith("__"))
     shelldeps = set(key for key in d.getVar("__exportlist", False) if d.getVarFlag(key, "export", False) and not d.getVarFlag(key, "unexport", False))
@@ -381,7 +383,7 @@ def generate_dependencies(d, whitelist):
         newdeps = deps[task]
         seen = set()
         while newdeps:
-            nextdeps = newdeps - whitelist
+            nextdeps = newdeps
             seen |= nextdeps
             newdeps = set()
             for dep in nextdeps:
@@ -392,43 +394,6 @@ def generate_dependencies(d, whitelist):
         #print "For %s: %s" % (task, str(deps[task]))
     return tasklist, deps, values
 
-def generate_dependency_hash(tasklist, gendeps, lookupcache, whitelist, fn):
-    taskdeps = {}
-    basehash = {}
-
-    for task in tasklist:
-        data = lookupcache[task]
-
-        if data is None:
-            bb.error("Task %s from %s seems to be empty?!" % (task, fn))
-            data = ''
-
-        gendeps[task] -= whitelist
-        newdeps = gendeps[task]
-        seen = set()
-        while newdeps:
-            nextdeps = newdeps
-            seen |= nextdeps
-            newdeps = set()
-            for dep in nextdeps:
-                if dep in whitelist:
-                    continue
-                gendeps[dep] -= whitelist
-                newdeps |= gendeps[dep]
-            newdeps -= seen
-
-        alldeps = sorted(seen)
-        for dep in alldeps:
-            data = data + dep
-            var = lookupcache[dep]
-            if var is not None:
-                data = data + str(var)
-        k = fn + ":" + task
-        basehash[k] = hashlib.sha256(data.encode("utf-8")).hexdigest()
-        taskdeps[task] = alldeps
-
-    return taskdeps, basehash
-
 def inherits_class(klass, d):
     val = d.getVar('__inherit_cache', False) or []
     needle = os.path.join('classes', '%s.bbclass' % klass)

bitbake/lib/bb/data_smart.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 """
 BitBake Smart Dictionary Implementation
 
@@ -12,12 +14,22 @@ BitBake build tools.
 # Copyright (C) 2005        Uli Luckas
 # Copyright (C) 2005        ROAD GmbH
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 # Based on functions from the base bb module, Copyright 2003 Holger Schurig
 
 import copy, re, sys, traceback
-from collections.abc import MutableMapping
+from collections import MutableMapping
 import logging
 import hashlib
 import bb, bb.codeparser
@@ -27,11 +39,9 @@ from bb.COW  import COWDictBase
 logger = logging.getLogger("BitBake.Data")
 
 __setvar_keyword__ = ["_append", "_prepend", "_remove"]
-__setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
-__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~:]+?}")
+__setvar_regexp__ = re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
+__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
 __expand_python_regexp__ = re.compile(r"\${@.+?}")
-__whitespace_split__ = re.compile(r'(\s)')
-__override_regexp__ = re.compile(r'[a-z0-9]+')
 
 def infer_caller_details(loginfo, parent = False, varval = True):
     """Save the caller the trouble of specifying everything."""
@@ -94,7 +104,11 @@ class VariableParse:
             if self.varname and key:
                 if self.varname == key:
                     raise Exception("variable %s references itself!" % self.varname)
-            var = self.d.getVarFlag(key, "_content")
+            if key in self.d.expand_cache:
+                varparse = self.d.expand_cache[key]
+                var = varparse.value
+            else:
+                var = self.d.getVarFlag(key, "_content")
             self.references.add(key)
             if var is not None:
                 return var
@@ -107,11 +121,11 @@ class VariableParse:
             else:
                 code = match.group()[3:-1]
 
-            if self.varname:
-                varname = 'Var <%s>' % self.varname
-            else:
-                varname = '<expansion>'
-            codeobj = compile(code.strip(), varname, "eval")
+            if "_remote_data" in self.d:
+                connector = self.d["_remote_data"]
+                return connector.expandPythonRef(self.varname, code, self.d)
+
+            codeobj = compile(code.strip(), self.varname or "<expansion>", "eval")
 
             parser = bb.codeparser.PythonParser(self.varname, logger)
             parser.parse_python(code)
@@ -253,18 +267,13 @@ class VariableHistory(object):
             return
         self.variables[var].append(loginfo.copy())
 
-    def rename_variable_hist(self, oldvar, newvar):
-        if not self.dataroot._tracking:
-            return
-        if oldvar not in self.variables:
-            return
-        if newvar not in self.variables:
-            self.variables[newvar] = []
-        for i in self.variables[oldvar]:
-            self.variables[newvar].append(i.copy())
-
     def variable(self, var):
-        varhistory = []
+        remote_connector = self.dataroot.getVar('_remote_data', False)
+        if remote_connector:
+            varhistory = remote_connector.getVarHistory(var)
+        else:
+            varhistory = []
+
         if var in self.variables:
             varhistory.extend(self.variables[var])
         return varhistory
@@ -329,12 +338,11 @@ class VariableHistory(object):
                 lines.append(line)
         return lines
 
-    def get_variable_items_files(self, var):
+    def get_variable_items_files(self, var, d):
         """
         Use variable history to map items added to a list variable and
         the files in which they were added.
         """
-        d = self.dataroot
         history = self.variable(var)
         finalitems = (d.getVar(var) or '').split()
         filemap = {}
@@ -393,6 +401,9 @@ class DataSmart(MutableMapping):
         if not isinstance(s, str): # sanity check
             return VariableParse(varname, self, s)
 
+        if varname and varname in self.expand_cache:
+            return self.expand_cache[varname]
+
         varparse = VariableParse(varname, self)
 
         while s.find('${') != -1:
@@ -403,7 +414,7 @@ class DataSmart(MutableMapping):
                     s = __expand_python_regexp__.sub(varparse.python_sub, s)
                 except SyntaxError as e:
                     # Likely unmatched brackets, just don't expand the expression
-                    if e.msg != "EOL while scanning string literal" and not e.msg.startswith("unterminated string literal"):
+                    if e.msg != "EOL while scanning string literal":
                         raise
                 if s == olds:
                     break
@@ -411,14 +422,14 @@ class DataSmart(MutableMapping):
                 raise
             except bb.parse.SkipRecipe:
                 raise
-            except bb.BBHandledException:
-                raise
             except Exception as exc:
-                tb = sys.exc_info()[2]
-                raise ExpansionError(varname, s, exc).with_traceback(tb) from exc
+                raise ExpansionError(varname, s, exc) from exc
 
         varparse.value = s
 
+        if varname:
+            self.expand_cache[varname] = varparse
+
         return varparse
 
     def expand(self, s, varname = None):
@@ -464,6 +475,10 @@ class DataSmart(MutableMapping):
             if var in dest:
                 return dest[var], self.overridedata.get(var, None)
 
+            if "_remote_data" in dest:
+                connector = dest["_remote_data"]["_content"]
+                return connector.getVar(var)
+
             if "_data" not in dest:
                 break
             dest = dest["_data"]
@@ -483,15 +498,19 @@ class DataSmart(MutableMapping):
 
     def setVar(self, var, value, **loginfo):
         #print("var=" + str(var) + "  val=" + str(value))
-        var = var.replace(":", "_")
-        self.expand_cache = {}
         parsing=False
         if 'parsing' in loginfo:
             parsing=True
 
+        if '_remote_data' in self.dict:
+            connector = self.dict["_remote_data"]["_content"]
+            res = connector.setVar(var, value)
+            if not res:
+                return
+
         if 'op' not in loginfo:
             loginfo['op'] = "set"
-
+        self.expand_cache = {}
         match  = __setvar_regexp__.match(var)
         if match and match.group("keyword") in __setvar_keyword__:
             base = match.group('base')
@@ -571,7 +590,7 @@ class DataSmart(MutableMapping):
         # aka pay the cookie monster
         override = var[var.rfind('_')+1:]
         shortvar = var[:var.rfind('_')]
-        while override and __override_regexp__.match(override):
+        while override and override.islower():
             if shortvar not in self.overridedata:
                 self.overridedata[shortvar] = []
             if [var, override] not in self.overridedata[shortvar]:
@@ -592,15 +611,14 @@ class DataSmart(MutableMapping):
         """
         Rename the variable key to newkey
         """
-        key = key.replace(":", "_")
-        newkey = newkey.replace(":", "_")
-        if key == newkey:
-            bb.warn("Calling renameVar with equivalent keys (%s) is invalid" % key)
-            return
+        if '_remote_data' in self.dict:
+            connector = self.dict["_remote_data"]["_content"]
+            res = connector.renameVar(key, newkey)
+            if not res:
+                return
 
         val = self.getVar(key, 0, parsing=True)
         if val is not None:
-            self.varhistory.rename_variable_hist(key, newkey)
             loginfo['variable'] = newkey
             loginfo['op'] = 'rename from %s' % key
             loginfo['detail'] = val
@@ -642,12 +660,16 @@ class DataSmart(MutableMapping):
         self.setVar(var + "_prepend", value, ignore=True, parsing=True)
 
     def delVar(self, var, **loginfo):
-        var = var.replace(":", "_")
-        self.expand_cache = {}
+        if '_remote_data' in self.dict:
+            connector = self.dict["_remote_data"]["_content"]
+            res = connector.delVar(var)
+            if not res:
+                return
 
         loginfo['detail'] = ""
         loginfo['op'] = 'del'
         self.varhistory.record(**loginfo)
+        self.expand_cache = {}
         self.dict[var] = {}
         if var in self.overridedata:
             del self.overridedata[var]
@@ -670,9 +692,13 @@ class DataSmart(MutableMapping):
                          override = None
 
     def setVarFlag(self, var, flag, value, **loginfo):
-        var = var.replace(":", "_")
-        self.expand_cache = {}
+        if '_remote_data' in self.dict:
+            connector = self.dict["_remote_data"]["_content"]
+            res = connector.setVarFlag(var, flag, value)
+            if not res:
+                return
 
+        self.expand_cache = {}
         if 'op' not in loginfo:
             loginfo['op'] = "set"
         loginfo['flag'] = flag
@@ -693,22 +719,9 @@ class DataSmart(MutableMapping):
                 self.dict["__exportlist"]["_content"] = set()
             self.dict["__exportlist"]["_content"].add(var)
 
-    def getVarFlag(self, var, flag, expand=True, noweakdefault=False, parsing=False, retparser=False):
-        var = var.replace(":", "_")
-        if flag == "_content":
-            cachename = var
-        else:
-            if not flag:
-                bb.warn("Calling getVarFlag with flag unset is invalid")
-                return None
-            cachename = var + "[" + flag + "]"
-
-        if expand and cachename in self.expand_cache:
-            return self.expand_cache[cachename].value
-
+    def getVarFlag(self, var, flag, expand=True, noweakdefault=False, parsing=False):
         local_var, overridedata = self._findVar(var)
         value = None
-        removes = set()
         if flag == "_content" and overridedata is not None and not parsing:
             match = False
             active = {}
@@ -735,11 +748,7 @@ class DataSmart(MutableMapping):
                             match = active[a]
                             del active[a]
             if match:
-                value, subparser = self.getVarFlag(match, "_content", False, retparser=True)
-                if hasattr(subparser, "removes"):
-                    # We have to carry the removes from the overridden variable to apply at the
-                    # end of processing
-                    removes = subparser.removes
+                value = self.getVar(match, False)
 
         if local_var is not None and value is None:
             if flag in local_var:
@@ -775,13 +784,17 @@ class DataSmart(MutableMapping):
                 if match:
                     value = r + value
 
-        parser = None
-        if expand or retparser:
-            parser = self.expandWithRefs(value, cachename)
-        if expand:
-            value = parser.value
+        if expand and value:
+            # Only getvar (flag == _content) hits the expand cache
+            cachename = None
+            if flag == "_content":
+                cachename = var
+            else:
+                cachename = var + "[" + flag + "]"
+            value = self.expand(value, cachename)
 
-        if value and flag == "_content" and local_var is not None and "_remove" in local_var and not parsing:
+        if value and flag == "_content" and local_var is not None and "_remove" in local_var:
+            removes = []
             self.need_overrides()
             for (r, o) in local_var["_remove"]:
                 match = True
@@ -790,41 +803,26 @@ class DataSmart(MutableMapping):
                         if not o2 in self.overrides:
                             match = False                            
                 if match:
-                    removes.add(r)
-
-        if value and flag == "_content" and not parsing:
-            if removes and parser:
-                expanded_removes = {}
-                for r in removes:
-                    expanded_removes[r] = self.expand(r).split()
-
-                parser.removes = set()
-                val = ""
-                for v in __whitespace_split__.split(parser.value):
-                    skip = False
-                    for r in removes:
-                        if v in expanded_removes[r]:
-                            parser.removes.add(r)
-                            skip = True
-                    if skip:
-                        continue
-                    val = val + v
-                parser.value = val
-                if expand:
-                    value = parser.value
-
-        if parser:
-            self.expand_cache[cachename] = parser
-
-        if retparser:
-            return value, parser
+                    removes.extend(self.expand(r).split())
 
+            if removes:
+                filtered = filter(lambda v: v not in removes,
+                                  value.split())
+                value = " ".join(filtered)
+                if expand and var in self.expand_cache:
+                    # We need to ensure the expand cache has the correct value
+                    # flag == "_content" here
+                    self.expand_cache[var].value = value
         return value
 
     def delVarFlag(self, var, flag, **loginfo):
-        var = var.replace(":", "_")
-        self.expand_cache = {}
+        if '_remote_data' in self.dict:
+            connector = self.dict["_remote_data"]["_content"]
+            res = connector.delVarFlag(var, flag)
+            if not res:
+                return
 
+        self.expand_cache = {}
         local_var, _ = self._findVar(var)
         if not local_var:
             return
@@ -840,7 +838,6 @@ class DataSmart(MutableMapping):
             del self.dict[var][flag]
 
     def appendVarFlag(self, var, flag, value, **loginfo):
-        var = var.replace(":", "_")
         loginfo['op'] = 'append'
         loginfo['flag'] = flag
         self.varhistory.record(**loginfo)
@@ -848,7 +845,6 @@ class DataSmart(MutableMapping):
         self.setVarFlag(var, flag, newvalue, ignore=True)
 
     def prependVarFlag(self, var, flag, value, **loginfo):
-        var = var.replace(":", "_")
         loginfo['op'] = 'prepend'
         loginfo['flag'] = flag
         self.varhistory.record(**loginfo)
@@ -856,7 +852,6 @@ class DataSmart(MutableMapping):
         self.setVarFlag(var, flag, newvalue, ignore=True)
 
     def setVarFlags(self, var, flags, **loginfo):
-        var = var.replace(":", "_")
         self.expand_cache = {}
         infer_caller_details(loginfo)
         if not var in self.dict:
@@ -871,7 +866,6 @@ class DataSmart(MutableMapping):
             self.dict[var][i] = flags[i]
 
     def getVarFlags(self, var, expand = False, internalflags=False):
-        var = var.replace(":", "_")
         local_var, _ = self._findVar(var)
         flags = {}
 
@@ -888,7 +882,6 @@ class DataSmart(MutableMapping):
 
 
     def delVarFlags(self, var, **loginfo):
-        var = var.replace(":", "_")
         self.expand_cache = {}
         if not var in self.dict:
             self._makeShadowCopy(var)
@@ -946,7 +939,7 @@ class DataSmart(MutableMapping):
 
     def localkeys(self):
         for key in self.dict:
-            if key not in ['_data']:
+            if key not in ['_data', '_remote_data']:
                 yield key
 
     def __iter__(self):
@@ -955,7 +948,7 @@ class DataSmart(MutableMapping):
         def keylist(d):        
             klist = set()
             for key in d:
-                if key in ["_data"]:
+                if key in ["_data", "_remote_data"]:
                     continue
                 if key in deleted:
                     continue
@@ -969,6 +962,13 @@ class DataSmart(MutableMapping):
             if "_data" in d:
                 klist |= keylist(d["_data"])
 
+            if "_remote_data" in d:
+                connector = d["_remote_data"]["_content"]
+                for key in connector.getKeys():
+                    if key in deleted:
+                        continue
+                    klist.add(key)
+
             return klist
 
         self.need_overrides()
@@ -1014,12 +1014,9 @@ class DataSmart(MutableMapping):
                 continue
 
             value = d.getVar(key, False) or ""
-            if type(value) is type(self):
-                data.update({key:value.get_hash()})
-            else:
-                data.update({key:value})
+            data.update({key:value})
 
-            varflags = d.getVarFlags(key, internalflags = True, expand=["vardepvalue"])
+            varflags = d.getVarFlags(key, internalflags = True)
             if not varflags:
                 continue
             for f in varflags:
@@ -1037,4 +1034,4 @@ class DataSmart(MutableMapping):
                     data.update({i:value})
 
         data_str = str([(k, data[k]) for k in sorted(data.keys())])
-        return hashlib.sha256(data_str.encode("utf-8")).hexdigest()
+        return hashlib.md5(data_str.encode("utf-8")).hexdigest()

bitbake/lib/bb/event.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 """
 BitBake 'Event' implementation
 
@@ -7,20 +9,31 @@ BitBake build tools.
 
 # Copyright (C) 2003, 2004  Chris Larson
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
-import ast
-import atexit
-import collections
-import logging
+import os, sys
+import warnings
 import pickle
-import sys
-import threading
+import logging
+import atexit
 import traceback
+import ast
+import threading
 
-import bb.exceptions
 import bb.utils
+import bb.compat
+import bb.exceptions
 
 # This is the pid for which we should generate the event. This is set when
 # the runqueue forks off.
@@ -56,7 +69,7 @@ def set_class_handlers(h):
     _handlers = h
 
 def clean_class_handlers():
-    return collections.OrderedDict()
+    return bb.compat.OrderedDict()
 
 # Internal
 _handlers = clean_class_handlers()
@@ -123,15 +136,11 @@ def fire_class_handlers(event, d):
 ui_queue = []
 @atexit.register
 def print_ui_queue():
-    global ui_queue
     """If we're exiting before a UI has been spawned, display any queued
     LogRecords to the console."""
     logger = logging.getLogger("BitBake")
     if not _uiready:
         from bb.msg import BBLogFormatter
-        # Flush any existing buffered content
-        sys.stdout.flush()
-        sys.stderr.flush()
         stdout = logging.StreamHandler(sys.stdout)
         stderr = logging.StreamHandler(sys.stderr)
         formatter = BBLogFormatter("%(levelname)s: %(message)s")
@@ -168,7 +177,6 @@ def print_ui_queue():
             logger.removeHandler(stderr)
         else:
             logger.removeHandler(stdout)
-        ui_queue = []
 
 def fire_ui_handlers(event, d):
     global _thread_lock
@@ -346,7 +354,7 @@ def set_UIHmask(handlerNum, level, debug_domains, mask):
 
 def getName(e):
     """Returns the name of a class or class instance"""
-    if getattr(e, "__name__", None) is None:
+    if getattr(e, "__name__", None) == None:
         return e.__class__.__name__
     else:
         return e.__name__
@@ -387,7 +395,7 @@ class RecipeEvent(Event):
         Event.__init__(self)
 
 class RecipePreFinalise(RecipeEvent):
-    """ Recipe Parsing Complete but not yet finalised"""
+    """ Recipe Parsing Complete but not yet finialised"""
 
 class RecipeTaskPreProcess(RecipeEvent):
     """
@@ -403,6 +411,23 @@ class RecipeTaskPreProcess(RecipeEvent):
 class RecipeParsed(RecipeEvent):
     """ Recipe Parsing Complete """
 
+class StampUpdate(Event):
+    """Trigger for any adjustment of the stamp files to happen"""
+
+    def __init__(self, targets, stampfns):
+        self._targets = targets
+        self._stampfns = stampfns
+        Event.__init__(self)
+
+    def getStampPrefix(self):
+        return self._stampfns
+
+    def getTargets(self):
+        return self._targets
+
+    stampPrefix = property(getStampPrefix)
+    targets = property(getTargets)
+
 class BuildBase(Event):
     """Base class for bitbake build events"""
 
@@ -424,6 +449,12 @@ class BuildBase(Event):
     def setName(self, name):
         self._name = name
 
+    def getCfg(self):
+        return self.data
+
+    def setCfg(self, cfg):
+        self.data = cfg
+
     def getFailures(self):
         """
         Return the number of failed packages
@@ -432,6 +463,9 @@ class BuildBase(Event):
 
     pkgs = property(getPkgs, setPkgs, None, "pkgs property")
     name = property(getName, setName, None, "name property")
+    cfg = property(getCfg, setCfg, None, "cfg property")
+
+
 
 class BuildInit(BuildBase):
     """buildFile or buildTargets was invoked"""
@@ -508,7 +542,7 @@ class NoProvider(Event):
         extra = ''
         if not self._reasons:
             if self._close_matches:
-                extra = ". Close matches:\n  %s" % '\n  '.join(sorted(set(self._close_matches)))
+                extra = ". Close matches:\n  %s" % '\n  '.join(self._close_matches)
 
         if self._dependees:
             msg = "Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)%s" % (r, self._item, ", ".join(self._dependees), r, extra)

bitbake/lib/bb/exceptions.py

@@ -1,6 +1,3 @@
-#
-# SPDX-License-Identifier: GPL-2.0-only
-#
 
 import inspect
 import traceback

bitbake/lib/bb/fetch2/__init__.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 """
 BitBake 'Fetch' implementations
 
@@ -8,7 +10,18 @@ BitBake build tools.
 # Copyright (C) 2003, 2004  Chris Larson
 # Copyright (C) 2012  Intel Corporation
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 #
 # Based on functions from the base bb module, Copyright 2003 Holger Schurig
 
@@ -33,9 +46,6 @@ _checksum_cache = bb.checksum.FileChecksumCache()
 
 logger = logging.getLogger("BitBake.Fetcher")
 
-CHECKSUM_LIST = [ "md5", "sha256", "sha1", "sha384", "sha512" ]
-SHOWN_CHECKSUM_LIST = ["sha256"]
-
 class BBFetchException(Exception):
     """Class all fetch exceptions inherit from"""
     def __init__(self, message):
@@ -134,9 +144,10 @@ class NonLocalMethod(Exception):
         Exception.__init__(self)
 
 class MissingChecksumEvent(bb.event.Event):
-    def __init__(self, url, **checksums):
+    def __init__(self, url, md5sum, sha256sum):
         self.url = url
-        self.checksums = checksums
+        self.checksums = {'md5sum': md5sum,
+                          'sha256sum': sha256sum}
         bb.event.Event.__init__(self)
 
 
@@ -245,7 +256,7 @@ class URI(object):
 
         # Identify if the URI is relative or not
         if urlp.scheme in self._relative_schemes and \
-           re.compile(r"^\w+:(?!//)").match(uri):
+           re.compile("^\w+:(?!//)").match(uri):
             self.relative = True
 
         if not self.relative:
@@ -372,7 +383,7 @@ def decodeurl(url):
         path = location
     else:
         host = location
-        path = "/"
+        path = ""
     if user:
         m = re.compile('(?P<user>[^:]+)(:?(?P<pswd>.*))').match(user)
         if m:
@@ -441,8 +452,8 @@ def uri_replace(ud, uri_find, uri_replace, replacements, d, mirrortarball=None):
             # Handle URL parameters
             if i:
                 # Any specified URL parameters must match
-                for k in uri_find_decoded[loc]:
-                    if uri_decoded[loc][k] != uri_find_decoded[loc][k]:
+                for k in uri_replace_decoded[loc]:
+                    if uri_decoded[loc][k] != uri_replace_decoded[loc][k]:
                         return None
             # Overwrite any specified replacement parameters
             for k in uri_replace_decoded[loc]:
@@ -486,22 +497,17 @@ def fetcher_init(d):
     Called to initialize the fetchers once the configuration data is known.
     Calls before this must not hit the cache.
     """
-
-    revs = bb.persist_data.persist('BB_URI_HEADREVS', d)
-    try:
-        # fetcher_init is called multiple times, so make sure we only save the
-        # revs the first time it is called.
-        if not bb.fetch2.saved_headrevs:
-            bb.fetch2.saved_headrevs = dict(revs)
-    except:
-        pass
-
     # When to drop SCM head revisions controlled by user policy
     srcrev_policy = d.getVar('BB_SRCREV_POLICY') or "clear"
     if srcrev_policy == "cache":
         logger.debug(1, "Keeping SRCREV cache due to cache policy of: %s", srcrev_policy)
     elif srcrev_policy == "clear":
         logger.debug(1, "Clearing SRCREV cache due to cache policy of: %s", srcrev_policy)
+        revs = bb.persist_data.persist('BB_URI_HEADREVS', d)
+        try:
+            bb.fetch2.saved_headrevs = revs.items()
+        except:
+            pass
         revs.clear()
     else:
         raise FetchError("Invalid SRCREV cache policy of: %s" % srcrev_policy)
@@ -518,14 +524,24 @@ def fetcher_parse_save():
 def fetcher_parse_done():
     _checksum_cache.save_merge()
 
-def fetcher_compare_revisions(d):
+def fetcher_compare_revisions():
     """
-    Compare the revisions in the persistent cache with the saved values from
-    when bitbake was started and return true if they have changed.
+    Compare the revisions in the persistant cache with current values and
+    return true/false on whether they've changed.
     """
 
-    headrevs = dict(bb.persist_data.persist('BB_URI_HEADREVS', d))
-    return headrevs != bb.fetch2.saved_headrevs
+    data = bb.persist_data.persist('BB_URI_HEADREVS', d).items()
+    data2 = bb.fetch2.saved_headrevs
+
+    changed = False
+    for key in data:
+        if key not in data2 or data2[key] != data[key]:
+            logger.debug(1, "%s changed", key)
+            changed = True
+            return True
+        else:
+            logger.debug(2, "%s did not change", key)
+    return False
 
 def mirror_from_string(data):
     mirrors = (data or "").replace('\\n',' ').split()
@@ -549,87 +565,71 @@ def verify_checksum(ud, d, precomputed={}):
     downloading. See https://bugzilla.yoctoproject.org/show_bug.cgi?id=5571.
     """
 
+    _MD5_KEY = "md5"
+    _SHA256_KEY = "sha256"
+
     if ud.ignore_checksums or not ud.method.supports_checksum(ud):
         return {}
 
-    def compute_checksum_info(checksum_id):
-        checksum_name = getattr(ud, "%s_name" % checksum_id)
+    if _MD5_KEY in precomputed:
+        md5data = precomputed[_MD5_KEY]
+    else:
+        md5data = bb.utils.md5_file(ud.localpath)
 
-        if checksum_id in precomputed:
-            checksum_data = precomputed[checksum_id]
-        else:
-            checksum_data = getattr(bb.utils, "%s_file" % checksum_id)(ud.localpath)
-
-        checksum_expected = getattr(ud, "%s_expected" % checksum_id)
-
-        if checksum_expected == '':
-            checksum_expected = None
-
-        return {
-            "id": checksum_id,
-            "name": checksum_name,
-            "data": checksum_data,
-            "expected": checksum_expected
-        }
-
-    checksum_infos = []
-    for checksum_id in CHECKSUM_LIST:
-        checksum_infos.append(compute_checksum_info(checksum_id))
-
-    checksum_dict = {ci["id"] : ci["data"] for ci in checksum_infos}
-    checksum_event = {"%ssum" % ci["id"] : ci["data"] for ci in checksum_infos}
-
-    for ci in checksum_infos:
-        if ci["id"] in SHOWN_CHECKSUM_LIST:
-            checksum_lines = ["SRC_URI[%s] = \"%s\"" % (ci["name"], ci["data"])]
-
-    # If no checksum has been provided
-    if ud.method.recommends_checksum(ud) and all(ci["expected"] is None for ci in checksum_infos):
-        messages = []
-        strict = d.getVar("BB_STRICT_CHECKSUM") or "0"
+    if _SHA256_KEY in precomputed:
+        sha256data = precomputed[_SHA256_KEY]
+    else:
+        sha256data = bb.utils.sha256_file(ud.localpath)
 
+    if ud.method.recommends_checksum(ud) and not ud.md5_expected and not ud.sha256_expected:
         # If strict checking enabled and neither sum defined, raise error
+        strict = d.getVar("BB_STRICT_CHECKSUM") or "0"
         if strict == "1":
-            messages.append("No checksum specified for '%s', please add at " \
-                            "least one to the recipe:" % ud.localpath)
-            messages.extend(checksum_lines)
-            logger.error("\n".join(messages))
-            raise NoChecksumError("Missing SRC_URI checksum", ud.url)
+            logger.error('No checksum specified for %s, please add at least one to the recipe:\n'
+                             'SRC_URI[%s] = "%s"\nSRC_URI[%s] = "%s"' %
+                             (ud.localpath, ud.md5_name, md5data,
+                              ud.sha256_name, sha256data))
+            raise NoChecksumError('Missing SRC_URI checksum', ud.url)
 
-        bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d)
+        bb.event.fire(MissingChecksumEvent(ud.url, md5data, sha256data), d)
 
         if strict == "ignore":
-            return checksum_dict
+            return {
+                _MD5_KEY: md5data,
+                _SHA256_KEY: sha256data
+            }
 
         # Log missing sums so user can more easily add them
-        messages.append("Missing checksum for '%s', consider adding at " \
-                        "least one to the recipe:" % ud.localpath)
-        messages.extend(checksum_lines)
-        logger.warning("\n".join(messages))
+        logger.warning('Missing md5 SRC_URI checksum for %s, consider adding to the recipe:\n'
+                       'SRC_URI[%s] = "%s"',
+                       ud.localpath, ud.md5_name, md5data)
+        logger.warning('Missing sha256 SRC_URI checksum for %s, consider adding to the recipe:\n'
+                       'SRC_URI[%s] = "%s"',
+                       ud.localpath, ud.sha256_name, sha256data)
 
     # We want to alert the user if a checksum is defined in the recipe but
     # it does not match.
-    messages = []
-    messages.append("Checksum mismatch!")
-    bad_checksum = None
+    msg = ""
+    mismatch = False
+    if ud.md5_expected and ud.md5_expected != md5data:
+        msg = msg + "\nFile: '%s' has %s checksum %s when %s was expected" % (ud.localpath, 'md5', md5data, ud.md5_expected)
+        mismatch = True;
 
-    for ci in checksum_infos:
-        if ci["expected"] and ci["expected"] != ci["data"]:
-            messages.append("File: '%s' has %s checksum '%s' when '%s' was " \
-                            "expected" % (ud.localpath, ci["id"], ci["data"], ci["expected"]))
-            bad_checksum = ci["data"]
+    if ud.sha256_expected and ud.sha256_expected != sha256data:
+        msg = msg + "\nFile: '%s' has %s checksum %s when %s was expected" % (ud.localpath, 'sha256', sha256data, ud.sha256_expected)
+        mismatch = True;
 
-    if bad_checksum:
-        messages.append("If this change is expected (e.g. you have upgraded " \
-                        "to a new version without updating the checksums) " \
-                        "then you can use these lines within the recipe:")
-        messages.extend(checksum_lines)
-        messages.append("Otherwise you should retry the download and/or " \
-                        "check with upstream to determine if the file has " \
-                        "become corrupted or otherwise unexpectedly modified.")
-        raise ChecksumError("\n".join(messages), ud.url, bad_checksum)
+    if mismatch:
+        msg = msg + '\nIf this change is expected (e.g. you have upgraded to a new version without updating the checksums) then you can use these lines within the recipe:\nSRC_URI[%s] = "%s"\nSRC_URI[%s] = "%s"\nOtherwise you should retry the download and/or check with upstream to determine if the file has become corrupted or otherwise unexpectedly modified.\n' % (ud.md5_name, md5data, ud.sha256_name, sha256data)
+
+    if len(msg):
+        raise ChecksumError('Checksum mismatch!%s' % msg, ud.url, md5data)
+
+    return {
+        _MD5_KEY: md5data,
+        _SHA256_KEY: sha256data
+    }
 
-    return checksum_dict
 
 def verify_donestamp(ud, d, origud=None):
     """
@@ -643,25 +643,26 @@ def verify_donestamp(ud, d, origud=None):
     if not ud.needdonestamp or (origud and not origud.needdonestamp):
         return True
 
-    if not os.path.exists(ud.localpath):
-        # local path does not exist
-        if os.path.exists(ud.donestamp):
-            # done stamp exists, but the downloaded file does not; the done stamp
-            # must be incorrect, re-trigger the download
-            bb.utils.remove(ud.donestamp)
+    if not os.path.exists(ud.donestamp):
         return False
 
     if (not ud.method.supports_checksum(ud) or
         (origud and not origud.method.supports_checksum(origud))):
-        # if done stamp exists and checksums not supported; assume the local
-        # file is current
-        return os.path.exists(ud.donestamp)
+        # done stamp exists, checksums not supported; assume the local file is
+        # current
+        return True
+
+    if not os.path.exists(ud.localpath):
+        # done stamp exists, but the downloaded file does not; the done stamp
+        # must be incorrect, re-trigger the download
+        bb.utils.remove(ud.donestamp)
+        return False
 
     precomputed_checksums = {}
     # Only re-use the precomputed checksums if the donestamp is newer than the
     # file. Do not rely on the mtime of directories, though. If ud.localpath is
     # a directory, there will probably not be any checksums anyway.
-    if os.path.exists(ud.donestamp) and (os.path.isdir(ud.localpath) or
+    if (os.path.isdir(ud.localpath) or
             os.path.getmtime(ud.localpath) < os.path.getmtime(ud.donestamp)):
         try:
             with open(ud.donestamp, "rb") as cachefile:
@@ -777,8 +778,7 @@ def get_srcrev(d, method_name='sortable_revision'):
     #
     format = d.getVar('SRCREV_FORMAT')
     if not format:
-        raise FetchError("The SRCREV_FORMAT variable must be set when multiple SCMs are used.\n"\
-                         "The SCMs are:\n%s" % '\n'.join(scms))
+        raise FetchError("The SRCREV_FORMAT variable must be set when multiple SCMs are used.")
 
     name_to_rev = {}
     seenautoinc = False
@@ -828,7 +828,6 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None):
                   'NO_PROXY', 'no_proxy',
                   'ALL_PROXY', 'all_proxy',
                   'GIT_PROXY_COMMAND',
-                  'GIT_SSH',
                   'GIT_SSL_CAINFO',
                   'GIT_SMART_HTTP',
                   'SSH_AUTH_SOCK', 'SSH_AGENT_PID',
@@ -839,16 +838,14 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None):
     if not cleanup:
         cleanup = []
 
-    # If PATH contains WORKDIR which contains PV-PR which contains SRCPV we
+    # If PATH contains WORKDIR which contains PV which contains SRCPV we
     # can end up in circular recursion here so give the option of breaking it
     # in a data store copy.
     try:
         d.getVar("PV")
-        d.getVar("PR")
     except bb.data_smart.ExpansionError:
         d = bb.data.createCopy(d)
         d.setVar("PV", "fetcheravoidrecurse")
-        d.setVar("PR", "fetcheravoidrecurse")
 
     origenv = d.getVar("BB_ORIGENV", False)
     for var in exportvars:
@@ -856,13 +853,7 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None):
         if val:
             cmd = 'export ' + var + '=\"%s\"; %s' % (val, cmd)
 
-    # Disable pseudo as it may affect ssh, potentially causing it to hang.
-    cmd = 'export PSEUDO_DISABLED=1; ' + cmd
-
-    if workdir:
-        logger.debug(1, "Running '%s' in %s" % (cmd, workdir))
-    else:
-        logger.debug(1, "Running %s", cmd)
+    logger.debug(1, "Running %s", cmd)
 
     success = False
     error_message = ""
@@ -898,7 +889,7 @@ def check_network_access(d, info, url):
     log remote network access, and error if BB_NO_NETWORK is set or the given
     URI is untrusted
     """
-    if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")):
+    if d.getVar("BB_NO_NETWORK") == "1":
         raise NetworkAccess(url, info)
     elif not trusted_network(d, url):
         raise UntrustedUrl(url, info)
@@ -970,8 +961,7 @@ def rename_bad_checksum(ud, suffix):
 
     new_localpath = "%s_bad-checksum_%s" % (ud.localpath, suffix)
     bb.warn("Renaming %s to %s" % (ud.localpath, new_localpath))
-    if not bb.utils.movefile(ud.localpath, new_localpath):
-        bb.warn("Renaming %s to %s failed, grep movefile in log.do_fetch to see why" % (ud.localpath, new_localpath))
+    bb.utils.movefile(ud.localpath, new_localpath)
 
 
 def try_mirror_url(fetch, origud, ud, ld, check = False):
@@ -1024,7 +1014,16 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
                     origud.method.build_mirror_data(origud, ld)
             return origud.localpath
         # Otherwise the result is a local file:// and we symlink to it
-        ensure_symlink(ud.localpath, origud.localpath)
+        if not os.path.exists(origud.localpath):
+            if os.path.islink(origud.localpath):
+                # Broken symbolic link
+                os.unlink(origud.localpath)
+
+            # As per above, in case two tasks end up here simultaneously.
+            try:
+                os.symlink(ud.localpath, origud.localpath)
+            except FileExistsError:
+                pass
         update_stamp(origud, ld)
         return ud.localpath
 
@@ -1032,7 +1031,7 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
         raise
 
     except IOError as e:
-        if e.errno in [errno.ESTALE]:
+        if e.errno in [os.errno.ESTALE]:
             logger.warning("Stale Error Observed %s." % ud.url)
             return False
         raise
@@ -1058,22 +1057,6 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
             bb.utils.unlockfile(lf)
 
 
-def ensure_symlink(target, link_name):
-    if not os.path.exists(link_name):
-        if os.path.islink(link_name):
-            # Broken symbolic link
-            os.unlink(link_name)
-
-        # In case this is executing without any file locks held (as is
-        # the case for file:// URLs), two tasks may end up here at the
-        # same time, in which case we do not want the second task to
-        # fail when the link has already been created by the first task.
-        try:
-            os.symlink(target, link_name)
-        except FileExistsError:
-            pass
-
-
 def try_mirrors(fetch, d, origud, mirrors, check = False):
     """
     Try to use a mirrored version of the sources.
@@ -1089,7 +1072,7 @@ def try_mirrors(fetch, d, origud, mirrors, check = False):
 
     for index, uri in enumerate(uris):
         ret = try_mirror_url(fetch, origud, uds[index], ld, check)
-        if ret:
+        if ret != False:
             return ret
     return None
 
@@ -1099,13 +1082,11 @@ def trusted_network(d, url):
     BB_ALLOWED_NETWORKS is set globally or for a specific recipe.
     Note: modifies SRC_URI & mirrors.
     """
-    if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")):
+    if d.getVar('BB_NO_NETWORK') == "1":
         return True
 
     pkgname = d.expand(d.getVar('PN', False))
-    trusted_hosts = None
-    if pkgname:
-        trusted_hosts = d.getVarFlag('BB_ALLOWED_NETWORKS', pkgname, False)
+    trusted_hosts = d.getVarFlag('BB_ALLOWED_NETWORKS', pkgname, False)
 
     if not trusted_hosts:
         trusted_hosts = d.getVar('BB_ALLOWED_NETWORKS')
@@ -1205,14 +1186,14 @@ def get_checksum_file_list(d):
 
     return " ".join(filelist)
 
-def get_file_checksums(filelist, pn, localdirsexclude):
+def get_file_checksums(filelist, pn):
     """Get a list of the checksums for a list of local files
 
     Returns the checksums for a list of local files, caching the results as
     it proceeds
 
     """
-    return _checksum_cache.get_checksums(filelist, pn, localdirsexclude)
+    return _checksum_cache.get_checksums(filelist, pn)
 
 
 class FetchData(object):
@@ -1238,26 +1219,24 @@ class FetchData(object):
             self.pswd = self.parm["pswd"]
         self.setup = False
 
-        def configure_checksum(checksum_id):
-            if "name" in self.parm:
-                checksum_name = "%s.%ssum" % (self.parm["name"], checksum_id)
-            else:
-                checksum_name = "%ssum" % checksum_id
-
-            setattr(self, "%s_name" % checksum_id, checksum_name)
-
-            if checksum_name in self.parm:
-                checksum_expected = self.parm[checksum_name]
-            elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3"]:
-                checksum_expected = None
-            else:
-                checksum_expected = d.getVarFlag("SRC_URI", checksum_name)
-
-            setattr(self, "%s_expected" % checksum_id, checksum_expected)
-
-        for checksum_id in CHECKSUM_LIST:
-            configure_checksum(checksum_id)
-
+        if "name" in self.parm:
+            self.md5_name = "%s.md5sum" % self.parm["name"]
+            self.sha256_name = "%s.sha256sum" % self.parm["name"]
+        else:
+            self.md5_name = "md5sum"
+            self.sha256_name = "sha256sum"
+        if self.md5_name in self.parm:
+            self.md5_expected = self.parm[self.md5_name]
+        elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3"]:
+            self.md5_expected = None
+        else:
+            self.md5_expected = d.getVarFlag("SRC_URI", self.md5_name)
+        if self.sha256_name in self.parm:
+            self.sha256_expected = self.parm[self.sha256_name]
+        elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3"]:
+            self.sha256_expected = None
+        else:
+            self.sha256_expected = d.getVarFlag("SRC_URI", self.sha256_name)
         self.ignore_checksums = False
 
         self.names = self.parm.get("name",'default').split(',')
@@ -1361,7 +1340,7 @@ class FetchMethod(object):
         """
 
         # We cannot compute checksums for directories
-        if os.path.isdir(urldata.localpath):
+        if os.path.isdir(urldata.localpath) == True:
             return False
         if urldata.localpath.find("*") != -1:
             return False
@@ -1375,18 +1354,6 @@ class FetchMethod(object):
         """
         return False
 
-    def verify_donestamp(self, ud, d):
-        """
-        Verify the donestamp file
-        """
-        return verify_donestamp(ud, d)
-
-    def update_donestamp(self, ud, d):
-        """
-        Update the donestamp file
-        """
-        update_stamp(ud, d)
-
     def _strip_leading_slashes(self, relpath):
         """
         Remove leading slash as os.path.join can't cope
@@ -1422,7 +1389,7 @@ class FetchMethod(object):
         Fetch urls
         Assumes localpath was called first
         """
-        raise NoMethodError(urldata.url)
+        raise NoMethodError(url)
 
     def unpack(self, urldata, rootdir, data):
         iterate = False
@@ -1457,7 +1424,7 @@ class FetchMethod(object):
                 cmd = 'gzip -dc %s > %s' % (file, efile)
             elif file.endswith('.bz2'):
                 cmd = 'bzip2 -dc %s > %s' % (file, efile)
-            elif file.endswith('.txz') or file.endswith('.tar.xz'):
+            elif file.endswith('.tar.xz'):
                 cmd = 'xz -dc %s | tar x --no-same-owner -f -' % file
             elif file.endswith('.xz'):
                 cmd = 'xz -dc %s > %s' % (file, efile)
@@ -1488,7 +1455,7 @@ class FetchMethod(object):
                 else:
                     cmd = 'rpm2cpio.sh %s | cpio -id' % (file)
             elif file.endswith('.deb') or file.endswith('.ipk'):
-                output = subprocess.check_output(['ar', '-t', file], preexec_fn=subprocess_setup)
+                output = subprocess.check_output('ar -t %s' % file, preexec_fn=subprocess_setup, shell=True)
                 datafile = None
                 if output:
                     for line in output.decode().splitlines():
@@ -1561,18 +1528,12 @@ class FetchMethod(object):
         """
         return True
 
-    def try_mirrors(self, fetch, urldata, d, mirrors, check=False):
-        """
-        Try to use a mirror
-        """
-        return bool(try_mirrors(fetch, d, urldata, mirrors, check))
-
     def checkstatus(self, fetch, urldata, d):
         """
         Check the status of a URL
         Assumes localpath was called first
         """
-        logger.info("URL %s could not be checked for status since no method exists.", urldata.url)
+        logger.info("URL %s could not be checked for status since no method exists.", url)
         return True
 
     def latest_revision(self, ud, d, name):
@@ -1580,7 +1541,7 @@ class FetchMethod(object):
         Look in the cache for the latest revision, if not present ask the SCM.
         """
         if not hasattr(self, "_latest_revision"):
-            raise ParameterError("The fetcher for this URL does not support _latest_revision", ud.url)
+            raise ParameterError("The fetcher for this URL does not support _latest_revision", url)
 
         revs = bb.persist_data.persist('BB_URI_HEADREVS', d)
         key = self.generate_revision_key(ud, d, name)
@@ -1595,7 +1556,8 @@ class FetchMethod(object):
         return True, str(latest_rev)
 
     def generate_revision_key(self, ud, d, name):
-        return self._revision_key(ud, d, name)
+        key = self._revision_key(ud, d, name)
+        return "%s-%s" % (key, d.getVar("PN") or "")
 
     def latest_versionstring(self, ud, d):
         """
@@ -1605,16 +1567,6 @@ class FetchMethod(object):
         """
         return ('', '')
 
-    def done(self, ud, d):
-        """
-        Is the download done ?
-        """
-        if os.path.exists(ud.localpath):
-            return True
-        if ud.localpath.find("*") != -1:
-            return True
-        return False
-
 class Fetch(object):
     def __init__(self, urls, d, cache = True, localonly = False, connection_cache = None):
         if localonly and cache:
@@ -1629,11 +1581,8 @@ class Fetch(object):
 
         fn = d.getVar('FILE')
         mc = d.getVar('__BBMULTICONFIG') or ""
-        key = None
-        if cache and fn:
-            key = mc + fn + str(id(d))
-        if key in urldata_cache:
-            self.ud = urldata_cache[key]
+        if cache and fn and mc + fn in urldata_cache:
+            self.ud = urldata_cache[mc + fn]
 
         for url in urls:
             if url not in self.ud:
@@ -1644,8 +1593,8 @@ class Fetch(object):
                         self.ud[url] = None
                         pass
 
-        if key:
-            urldata_cache[key] = self.ud
+        if fn and cache:
+            urldata_cache[mc + fn] = self.ud
 
     def localpath(self, url):
         if url not in self.urls:
@@ -1675,13 +1624,13 @@ class Fetch(object):
             urls = self.urls
 
         network = self.d.getVar("BB_NO_NETWORK")
-        premirroronly = bb.utils.to_boolean(self.d.getVar("BB_FETCH_PREMIRRORONLY"))
+        premirroronly = (self.d.getVar("BB_FETCH_PREMIRRORONLY") == "1")
 
         for u in urls:
             ud = self.ud[u]
             ud.setup_localpath(self.d)
             m = ud.method
-            done = False
+            localpath = ""
 
             if ud.lockfile:
                 lf = bb.utils.lockfile(ud.lockfile)
@@ -1689,28 +1638,28 @@ class Fetch(object):
             try:
                 self.d.setVar("BB_NO_NETWORK", network)
 
-                if m.verify_donestamp(ud, self.d) and not m.need_update(ud, self.d):
-                    done = True
+                if verify_donestamp(ud, self.d) and not m.need_update(ud, self.d):
+                    localpath = ud.localpath
                 elif m.try_premirror(ud, self.d):
                     logger.debug(1, "Trying PREMIRRORS")
                     mirrors = mirror_from_string(self.d.getVar('PREMIRRORS'))
-                    done = m.try_mirrors(self, ud, self.d, mirrors)
-                    if done:
+                    localpath = try_mirrors(self, self.d, ud, mirrors, False)
+                    if localpath:
                         try:
                             # early checksum verification so that if the checksum of the premirror
                             # contents mismatch the fetcher can still try upstream and mirrors
-                            m.update_donestamp(ud, self.d)
+                            update_stamp(ud, self.d)
                         except ChecksumError as e:
                             logger.warning("Checksum failure encountered with premirror download of %s - will attempt other sources." % u)
                             logger.debug(1, str(e))
-                            done = False
+                            localpath = ""
 
                 if premirroronly:
                     self.d.setVar("BB_NO_NETWORK", "1")
 
                 firsterr = None
-                verified_stamp = m.verify_donestamp(ud, self.d)
-                if not done and (not verified_stamp or m.need_update(ud, self.d)):
+                verified_stamp = verify_donestamp(ud, self.d)
+                if not localpath and (not verified_stamp or m.need_update(ud, self.d)):
                     try:
                         if not trusted_network(self.d, ud.url):
                             raise UntrustedUrl(ud.url)
@@ -1718,10 +1667,10 @@ class Fetch(object):
                         m.download(ud, self.d)
                         if hasattr(m, "build_mirror_data"):
                             m.build_mirror_data(ud, self.d)
-                        done = True
+                        localpath = ud.localpath
                         # early checksum verify, so that if checksum mismatched,
                         # fetcher still have chance to fetch from mirror
-                        m.update_donestamp(ud, self.d)
+                        update_stamp(ud, self.d)
 
                     except bb.fetch2.NetworkAccess:
                         raise
@@ -1743,17 +1692,17 @@ class Fetch(object):
                             m.clean(ud, self.d)
                         logger.debug(1, "Trying MIRRORS")
                         mirrors = mirror_from_string(self.d.getVar('MIRRORS'))
-                        done = m.try_mirrors(self, ud, self.d, mirrors)
+                        localpath = try_mirrors(self, self.d, ud, mirrors)
 
-                if not done or not m.done(ud, self.d):
+                if not localpath or ((not os.path.exists(localpath)) and localpath.find("*") == -1):
                     if firsterr:
                         logger.error(str(firsterr))
                     raise FetchError("Unable to fetch URL from any source.", u)
 
-                m.update_donestamp(ud, self.d)
+                update_stamp(ud, self.d)
 
             except IOError as e:
-                if e.errno in [errno.ESTALE]:
+                if e.errno in [os.errno.ESTALE]:
                     logger.error("Stale Error Observed %s." % u)
                     raise ChecksumError("Stale Error Detected")
 
@@ -1781,14 +1730,14 @@ class Fetch(object):
             logger.debug(1, "Testing URL %s", u)
             # First try checking uri, u, from PREMIRRORS
             mirrors = mirror_from_string(self.d.getVar('PREMIRRORS'))
-            ret = m.try_mirrors(self, ud, self.d, mirrors, True)
+            ret = try_mirrors(self, self.d, ud, mirrors, True)
             if not ret:
                 # Next try checking from the original uri, u
                 ret = m.checkstatus(self, ud, self.d)
                 if not ret:
                     # Finally, try checking uri, u, from MIRRORS
                     mirrors = mirror_from_string(self.d.getVar('MIRRORS'))
-                    ret = m.try_mirrors(self, ud, self.d, mirrors, True)
+                    ret = try_mirrors(self, self.d, ud, mirrors, True)
 
             if not ret:
                 raise FetchError("URL %s doesn't work" % u, u)
@@ -1823,7 +1772,7 @@ class Fetch(object):
 
         for url in urls:
             if url not in self.ud:
-                self.ud[url] = FetchData(url, self.d)
+                self.ud[url] = FetchData(url, d)
             ud = self.ud[url]
             ud.setup_localpath(self.d)
 
@@ -1893,7 +1842,6 @@ from . import osc
 from . import repo
 from . import clearcase
 from . import npm
-from . import npmsw
 
 methods.append(local.Local())
 methods.append(wget.Wget())
@@ -1912,4 +1860,3 @@ methods.append(osc.Osc())
 methods.append(repo.Repo())
 methods.append(clearcase.ClearCase())
 methods.append(npm.Npm())
-methods.append(npmsw.NpmShrinkWrap())

bitbake/lib/bb/fetch2/bzr.py

@@ -10,10 +10,22 @@ BitBake 'Fetch' implementation for bzr.
 #   BitBake build tools.
 #   Copyright (C) 2003, 2004  Chris Larson
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import os
+import sys
+import logging
 import bb
 from bb.fetch2 import FetchMethod
 from bb.fetch2 import FetchError
@@ -29,9 +41,8 @@ class Bzr(FetchMethod):
         init bzr specific variable within url data
         """
         # Create paths to bzr checkouts
-        bzrdir = d.getVar("BZRDIR") or (d.getVar("DL_DIR") + "/bzr")
         relpath = self._strip_leading_slashes(ud.path)
-        ud.pkgdir = os.path.join(bzrdir, ud.host, relpath)
+        ud.pkgdir = os.path.join(d.expand('${BZRDIR}'), ud.host, relpath)
 
         ud.setup_revisions(d)
 
@@ -46,7 +57,7 @@ class Bzr(FetchMethod):
         command is "fetch", "update", "revno"
         """
 
-        basecmd = d.getVar("FETCHCMD_bzr") or "/usr/bin/env bzr"
+        basecmd = d.expand('${FETCHCMD_bzr}')
 
         proto =  ud.parm.get('protocol', 'http')
 

bitbake/lib/bb/fetch2/clearcase.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 """
 BitBake 'Fetch' clearcase implementation
 
@@ -45,18 +47,29 @@ User credentials:
 """
 # Copyright (C) 2014 Siemens AG
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 #
 
 import os
+import sys
 import shutil
 import bb
 from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import FetchError
-from   bb.fetch2 import MissingParameterError
-from   bb.fetch2 import ParameterError
 from   bb.fetch2 import runfetchcmd
 from   bb.fetch2 import logger
+from   distutils import spawn
 
 class ClearCase(FetchMethod):
     """Class to fetch urls via 'clearcase'"""
@@ -80,7 +93,7 @@ class ClearCase(FetchMethod):
         if 'protocol' in ud.parm:
             ud.proto = ud.parm['protocol']
         if not ud.proto in ('http', 'https'):
-            raise ParameterError("Invalid protocol type", ud.url)
+            raise fetch2.ParameterError("Invalid protocol type", ud.url)
 
         ud.vob = ''
         if 'vob' in ud.parm:
@@ -94,7 +107,7 @@ class ClearCase(FetchMethod):
         else:
             ud.module = ""
 
-        ud.basecmd = d.getVar("FETCHCMD_ccrc") or "/usr/bin/env cleartool || rcleartool"
+        ud.basecmd = d.getVar("FETCHCMD_ccrc") or spawn.find_executable("cleartool") or spawn.find_executable("rcleartool")
 
         if d.getVar("SRCREV") == "INVALID":
           raise FetchError("Set a valid SRCREV for the clearcase fetcher in your recipe, e.g. SRCREV = \"/main/LATEST\" or any other label of your choice.")
@@ -144,18 +157,18 @@ class ClearCase(FetchMethod):
 
         basecmd = "%s %s" % (ud.basecmd, command)
 
-        if command == 'mkview':
+        if command is 'mkview':
             if not "rcleartool" in ud.basecmd:
                 # Cleartool needs a -snapshot view
                 options.append("-snapshot")
             options.append("-tag %s" % ud.viewname)
             options.append(ud.viewdir)
 
-        elif command == 'rmview':
+        elif command is 'rmview':
             options.append("-force")
             options.append("%s" % ud.viewdir)
 
-        elif command == 'setcs':
+        elif command is 'setcs':
             options.append("-overwrite")
             options.append(ud.configspecfile)
 
@@ -237,7 +250,7 @@ class ClearCase(FetchMethod):
 
         # Clean clearcase meta-data before tar
 
-        runfetchcmd('tar -czf "%s" .' % (ud.localpath), d, cleanup = [ud.localpath], workdir = ud.viewdir)
+        runfetchcmd('tar -czf "%s" .' % (ud.localpath), d, cleanup = [ud.localpath])
 
         # Clean up so we can create a new view next time
         self.clean(ud, d);

bitbake/lib/bb/fetch2/cvs.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 """
 BitBake 'Fetch' implementations
 
@@ -8,12 +10,24 @@ BitBake build tools.
 
 # Copyright (C) 2003, 2004  Chris Larson
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
-# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+#Based on functions from the base bb module, Copyright 2003 Holger Schurig
 #
 
 import os
+import logging
 import bb
 from bb.fetch2 import FetchMethod, FetchError, MissingParameterError, logger
 from bb.fetch2 import runfetchcmd
@@ -96,7 +110,7 @@ class Cvs(FetchMethod):
         if ud.tag:
             options.append("-r %s" % ud.tag)
 
-        cvsbasecmd = d.getVar("FETCHCMD_cvs") or "/usr/bin/env cvs"
+        cvsbasecmd = d.getVar("FETCHCMD_cvs")
         cvscmd = cvsbasecmd + " '-d" + cvsroot + "' co " + " ".join(options) + " " + ud.module
         cvsupdatecmd = cvsbasecmd + " '-d" + cvsroot + "' update -d -P " + " ".join(options)
 
@@ -107,8 +121,7 @@ class Cvs(FetchMethod):
         # create module directory
         logger.debug(2, "Fetch: checking for module directory")
         pkg = d.getVar('PN')
-        cvsdir = d.getVar("CVSDIR") or (d.getVar("DL_DIR") + "/cvs")
-        pkgdir = os.path.join(cvsdir, pkg)
+        pkgdir = os.path.join(d.getVar('CVSDIR'), pkg)
         moddir = os.path.join(pkgdir, localdir)
         workdir = None
         if os.access(os.path.join(moddir, 'CVS'), os.R_OK):

bitbake/lib/bb/fetch2/git.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 """
 BitBake 'Fetch' git implementation
 
@@ -44,8 +46,7 @@ Supported SRC_URI options are:
 
 - nobranch
    Don't check the SHA validation for branch. set this option for the recipe
-   referring to commit which is valid in any namespace (branch, tag, ...)
-   instead of branch.
+   referring to commit which is valid in tag instead of branch.
    The default is "0", set nobranch=1 if needed.
 
 - usehead
@@ -54,22 +55,30 @@ Supported SRC_URI options are:
 
 """
 
-# Copyright (C) 2005 Richard Purdie
+#Copyright (C) 2005 Richard Purdie
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import collections
 import errno
 import fnmatch
 import os
 import re
-import shlex
 import subprocess
 import tempfile
 import bb
 import bb.progress
-from contextlib import contextmanager
 from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import runfetchcmd
 from   bb.fetch2 import logger
@@ -116,9 +125,6 @@ class GitProgressHandler(bb.progress.LineFilterProgressHandler):
 
 
 class Git(FetchMethod):
-    bitbake_dir = os.path.abspath(os.path.join(os.path.dirname(os.path.join(os.path.abspath(__file__))), '..', '..', '..'))
-    make_shallow_path = os.path.join(bitbake_dir, 'bin', 'git-make-shallow')
-
     """Class to fetch a module or modules from git repositories"""
     def init(self, d):
         pass
@@ -143,10 +149,6 @@ class Git(FetchMethod):
             ud.proto = 'file'
         else:
             ud.proto = "git"
-        if ud.host == "github.com" and ud.proto == "git":
-            # github stopped supporting git protocol
-            # https://github.blog/2021-09-01-improving-git-protocol-security-github/#no-more-unauthenticated-git
-            ud.proto = "https"
 
         if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'):
             raise bb.fetch2.ParameterError("Invalid protocol type", ud.url)
@@ -194,7 +196,7 @@ class Git(FetchMethod):
             depth_default = 1
         ud.shallow_depths = collections.defaultdict(lambda: depth_default)
 
-        revs_default = d.getVar("BB_GIT_SHALLOW_REVS")
+        revs_default = d.getVar("BB_GIT_SHALLOW_REVS", True)
         ud.shallow_revs = []
         ud.branches = {}
         for pos, name in enumerate(ud.names):
@@ -226,12 +228,7 @@ class Git(FetchMethod):
             ud.shallow = False
 
         if ud.usehead:
-            # When usehead is set let's associate 'HEAD' with the unresolved
-            # rev of this repository. This will get resolved into a revision
-            # later. If an actual revision happens to have also been provided
-            # then this setting will be overridden.
-            for name in ud.names:
-                ud.unresolvedrev[name] = 'HEAD'
+            ud.unresolvedrev['default'] = 'HEAD'
 
         ud.basecmd = d.getVar("FETCHCMD_git") or "git -c core.fsyncobjectfiles=0"
 
@@ -248,7 +245,7 @@ class Git(FetchMethod):
                     ud.unresolvedrev[name] = ud.revisions[name]
                 ud.revisions[name] = self.latest_revision(ud, d, name)
 
-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_'))
+        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.'))
         if gitsrcname.startswith('.'):
             gitsrcname = gitsrcname[1:]
 
@@ -261,7 +258,7 @@ class Git(FetchMethod):
                 gitsrcname = gitsrcname + '_' + ud.revisions[name]
 
         dl_dir = d.getVar("DL_DIR")
-        gitdir = d.getVar("GITDIR") or (dl_dir + "/git2")
+        gitdir = d.getVar("GITDIR") or (dl_dir + "/git2/")
         ud.clonedir = os.path.join(gitdir, gitsrcname)
         ud.localfile = ud.clonedir
 
@@ -299,36 +296,21 @@ class Git(FetchMethod):
         return ud.clonedir
 
     def need_update(self, ud, d):
-        return self.clonedir_need_update(ud, d) or self.shallow_tarball_need_update(ud) or self.tarball_need_update(ud)
-
-    def clonedir_need_update(self, ud, d):
         if not os.path.exists(ud.clonedir):
             return True
-        if ud.shallow and ud.write_shallow_tarballs and self.clonedir_need_shallow_revs(ud, d):
-            return True
         for name in ud.names:
             if not self._contains_ref(ud, d, name, ud.clonedir):
                 return True
+        if ud.shallow and ud.write_shallow_tarballs and not os.path.exists(ud.fullshallow):
+            return True
+        if ud.write_tarballs and not os.path.exists(ud.fullmirror):
+            return True
         return False
 
-    def clonedir_need_shallow_revs(self, ud, d):
-        for rev in ud.shallow_revs:
-            try:
-                runfetchcmd('%s rev-parse -q --verify %s' % (ud.basecmd, rev), d, quiet=True, workdir=ud.clonedir)
-            except bb.fetch2.FetchError:
-                return rev
-        return None
-
-    def shallow_tarball_need_update(self, ud):
-        return ud.shallow and ud.write_shallow_tarballs and not os.path.exists(ud.fullshallow)
-
-    def tarball_need_update(self, ud):
-        return ud.write_tarballs and not os.path.exists(ud.fullmirror)
-
     def try_premirror(self, ud, d):
         # If we don't do this, updating an existing checkout with only premirrors
         # is not possible
-        if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")):
+        if d.getVar("BB_FETCH_PREMIRRORONLY") is not None:
             return True
         if os.path.exists(ud.clonedir):
             return False
@@ -337,13 +319,16 @@ class Git(FetchMethod):
     def download(self, ud, d):
         """Fetch url"""
 
+        no_clone = not os.path.exists(ud.clonedir)
+        need_update = no_clone or self.need_update(ud, d)
+
         # A current clone is preferred to either tarball, a shallow tarball is
         # preferred to an out of date clone, and a missing clone will use
         # either tarball.
-        if ud.shallow and os.path.exists(ud.fullshallow) and self.need_update(ud, d):
+        if ud.shallow and os.path.exists(ud.fullshallow) and need_update:
             ud.localpath = ud.fullshallow
             return
-        elif os.path.exists(ud.fullmirror) and not os.path.exists(ud.clonedir):
+        elif os.path.exists(ud.fullmirror) and no_clone:
             bb.utils.mkdirhier(ud.clonedir)
             runfetchcmd("tar -xzf %s" % ud.fullmirror, d, workdir=ud.clonedir)
 
@@ -354,90 +339,41 @@ class Git(FetchMethod):
             # We do this since git will use a "-l" option automatically for local urls where possible
             if repourl.startswith("file://"):
                 repourl = repourl[7:]
-            clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, shlex.quote(repourl), ud.clonedir)
+            clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, repourl, ud.clonedir)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, clone_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
             runfetchcmd(clone_cmd, d, log=progresshandler)
 
         # Update the checkout if needed
-        if self.clonedir_need_update(ud, d):
-            output = runfetchcmd("%s remote" % ud.basecmd, d, quiet=True, workdir=ud.clonedir)
-            if "origin" in output:
-              runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
+        needupdate = False
+        for name in ud.names:
+            if not self._contains_ref(ud, d, name, ud.clonedir):
+                needupdate = True
+        if needupdate:
+            try: 
+                runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
+            except bb.fetch2.FetchError:
+                logger.debug(1, "No Origin")
 
-            runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir)
-
-            if ud.nobranch:
-                fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl))
-            else:
-                fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*" % (ud.basecmd, shlex.quote(repourl))
+            runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, repourl), d, workdir=ud.clonedir)
+            fetch_cmd = "LANG=C %s fetch -f --prune --progress %s refs/*:refs/*" % (ud.basecmd, repourl)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
             runfetchcmd(fetch_cmd, d, log=progresshandler, workdir=ud.clonedir)
             runfetchcmd("%s prune-packed" % ud.basecmd, d, workdir=ud.clonedir)
-            runfetchcmd("%s pack-refs --all" % ud.basecmd, d, workdir=ud.clonedir)
             runfetchcmd("%s pack-redundant --all | xargs -r rm" % ud.basecmd, d, workdir=ud.clonedir)
             try:
                 os.unlink(ud.fullmirror)
             except OSError as exc:
                 if exc.errno != errno.ENOENT:
                     raise
-
         for name in ud.names:
             if not self._contains_ref(ud, d, name, ud.clonedir):
                 raise bb.fetch2.FetchError("Unable to find revision %s in branch %s even from upstream" % (ud.revisions[name], ud.branches[name]))
 
-        if ud.shallow and ud.write_shallow_tarballs:
-            missing_rev = self.clonedir_need_shallow_revs(ud, d)
-            if missing_rev:
-                raise bb.fetch2.FetchError("Unable to find revision %s even from upstream" % missing_rev)
-
-        if self._contains_lfs(ud, d, ud.clonedir) and self._need_lfs(ud):
-            # Unpack temporary working copy, use it to run 'git checkout' to force pre-fetching
-            # of all LFS blobs needed at the the srcrev.
-            #
-            # It would be nice to just do this inline here by running 'git-lfs fetch'
-            # on the bare clonedir, but that operation requires a working copy on some
-            # releases of Git LFS.
-            tmpdir = tempfile.mkdtemp(dir=d.getVar('DL_DIR'))
-            try:
-                # Do the checkout. This implicitly involves a Git LFS fetch.
-                Git.unpack(self, ud, tmpdir, d)
-
-                # Scoop up a copy of any stuff that Git LFS downloaded. Merge them into
-                # the bare clonedir.
-                #
-                # As this procedure is invoked repeatedly on incremental fetches as
-                # a recipe's SRCREV is bumped throughout its lifetime, this will
-                # result in a gradual accumulation of LFS blobs in <ud.clonedir>/lfs
-                # corresponding to all the blobs reachable from the different revs
-                # fetched across time.
-                #
-                # Only do this if the unpack resulted in a .git/lfs directory being
-                # created; this only happens if at least one blob needed to be
-                # downloaded.
-                if os.path.exists(os.path.join(tmpdir, "git", ".git", "lfs")):
-                    runfetchcmd("tar -cf - lfs | tar -xf - -C %s" % ud.clonedir, d, workdir="%s/git/.git" % tmpdir)
-            finally:
-                bb.utils.remove(tmpdir, recurse=True)
-
     def build_mirror_data(self, ud, d):
-
-        # Create as a temp file and move atomically into position to avoid races
-        @contextmanager
-        def create_atomic(filename):
-            fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename))
-            try:
-                yield tfile
-                umask = os.umask(0o666)
-                os.umask(umask)
-                os.chmod(tfile, (0o666 & ~umask))
-                os.rename(tfile, filename)
-            finally:
-                os.close(fd)
-
         if ud.shallow and ud.write_shallow_tarballs:
             if not os.path.exists(ud.fullshallow):
                 if os.path.islink(ud.fullshallow):
@@ -448,8 +384,7 @@ class Git(FetchMethod):
                     self.clone_shallow_local(ud, shallowclone, d)
 
                     logger.info("Creating tarball of git repository")
-                    with create_atomic(ud.fullshallow) as tfile:
-                        runfetchcmd("tar -czf %s ." % tfile, d, workdir=shallowclone)
+                    runfetchcmd("tar -czf %s ." % ud.fullshallow, d, workdir=shallowclone)
                     runfetchcmd("touch %s.done" % ud.fullshallow, d)
                 finally:
                     bb.utils.remove(tempdir, recurse=True)
@@ -458,8 +393,7 @@ class Git(FetchMethod):
                 os.unlink(ud.fullmirror)
 
             logger.info("Creating tarball of git repository")
-            with create_atomic(ud.fullmirror) as tfile:
-                runfetchcmd("tar -czf %s ." % tfile, d, workdir=ud.clonedir)
+            runfetchcmd("tar -czf %s ." % ud.fullmirror, d, workdir=ud.clonedir)
             runfetchcmd("touch %s.done" % ud.fullmirror, d)
 
     def clone_shallow_local(self, ud, dest, d):
@@ -511,7 +445,7 @@ class Git(FetchMethod):
                 shallow_branches.append(r)
 
         # Make the repository shallow
-        shallow_cmd = [self.make_shallow_path, '-s']
+        shallow_cmd = ['git', 'make-shallow', '-s']
         for b in shallow_branches:
             shallow_cmd.append('-r')
             shallow_cmd.append(b)
@@ -534,45 +468,14 @@ class Git(FetchMethod):
         if os.path.exists(destdir):
             bb.utils.prunedir(destdir)
 
-        need_lfs = self._need_lfs(ud)
-
-        if not need_lfs:
-            ud.basecmd = "GIT_LFS_SKIP_SMUDGE=1 " + ud.basecmd
-
-        source_found = False
-        source_error = []
-
-        if not source_found:
-            clonedir_is_up_to_date = not self.clonedir_need_update(ud, d)
-            if clonedir_is_up_to_date:
-                runfetchcmd("%s clone %s %s/ %s" % (ud.basecmd, ud.cloneflags, ud.clonedir, destdir), d)
-                source_found = True
-            else:
-                source_error.append("clone directory not available or not up to date: " + ud.clonedir)
-
-        if not source_found:
-            if ud.shallow:
-                if os.path.exists(ud.fullshallow):
-                    bb.utils.mkdirhier(destdir)
-                    runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=destdir)
-                    source_found = True
-                else:
-                    source_error.append("shallow clone not available: " + ud.fullshallow)
-            else:
-                source_error.append("shallow clone not enabled")
-
-        if not source_found:
-            raise bb.fetch2.UnpackError("No up to date source found: " + "; ".join(source_error), ud.url)
+        if ud.shallow and (not os.path.exists(ud.clonedir) or self.need_update(ud, d)):
+            bb.utils.mkdirhier(destdir)
+            runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=destdir)
+        else:
+            runfetchcmd("%s clone %s %s/ %s" % (ud.basecmd, ud.cloneflags, ud.clonedir, destdir), d)
 
         repourl = self._get_repo_url(ud)
-        runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=destdir)
-
-        if self._contains_lfs(ud, d, destdir):
-            if need_lfs and not self._find_git_lfs(d):
-                raise bb.fetch2.FetchError("Repository %s has LFS content, install git-lfs on host to download (or set lfs=0 to ignore it)" % (repourl))
-            elif not need_lfs:
-                bb.note("Repository %s has LFS content but it is not being fetched" % (repourl))
-
+        runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, repourl), d, workdir=destdir)
         if not ud.nocheckout:
             if subdir != "":
                 runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.revisions[ud.names[0]], readpathspec), d,
@@ -592,17 +495,9 @@ class Git(FetchMethod):
     def clean(self, ud, d):
         """ clean the git directory """
 
-        to_remove = [ud.localpath, ud.fullmirror, ud.fullmirror + ".done"]
-        # The localpath is a symlink to clonedir when it is cloned from a
-        # mirror, so remove both of them.
-        if os.path.islink(ud.localpath):
-            clonedir = os.path.realpath(ud.localpath)
-            to_remove.append(clonedir)
-
-        for r in to_remove:
-            if os.path.exists(r):
-                bb.note('Removing %s' % r)
-                bb.utils.remove(r, True)
+        bb.utils.remove(ud.localpath, True)
+        bb.utils.remove(ud.fullmirror)
+        bb.utils.remove(ud.fullmirror + ".done")
 
     def supports_srcrev(self):
         return True
@@ -623,43 +518,6 @@ class Git(FetchMethod):
             raise bb.fetch2.FetchError("The command '%s' gave output with more then 1 line unexpectedly, output: '%s'" % (cmd, output))
         return output.split()[0] != "0"
 
-    def _need_lfs(self, ud):
-        return ud.parm.get("lfs", "1") == "1"
-
-    def _contains_lfs(self, ud, d, wd):
-        """
-        Check if the repository has 'lfs' (large file) content
-        """
-
-        if not ud.nobranch:
-            branchname = ud.branches[ud.names[0]]
-        else:
-            branchname = "master"
-
-        # The bare clonedir doesn't use the remote names; it has the branch immediately.
-        if wd == ud.clonedir:
-            refname = ud.branches[ud.names[0]]
-        else:
-            refname = "origin/%s" % ud.branches[ud.names[0]]
-
-        cmd = "%s grep lfs %s:.gitattributes | wc -l" % (
-            ud.basecmd, refname)
-
-        try:
-            output = runfetchcmd(cmd, d, quiet=True, workdir=wd)
-            if int(output) > 0:
-                return True
-        except (bb.fetch2.FetchError,ValueError):
-            pass
-        return False
-
-    def _find_git_lfs(self, d):
-        """
-        Return True if git-lfs can be found, False otherwise.
-        """
-        import shutil
-        return shutil.which("git-lfs", path=d.getVar('PATH')) is not None
-
     def _get_repo_url(self, ud):
         """
         Return the repository URL
@@ -674,9 +532,7 @@ class Git(FetchMethod):
         """
         Return a unique key for the url
         """
-        # Collapse adjacent slashes
-        slash_re = re.compile(r"/+")
-        return "git:" + ud.host + slash_re.sub(".", ud.path) + ud.unresolvedrev[name]
+        return "git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name]
 
     def _lsremote(self, ud, d, search):
         """
@@ -694,7 +550,7 @@ class Git(FetchMethod):
         try:
             repourl = self._get_repo_url(ud)
             cmd = "%s ls-remote %s %s" % \
-                (ud.basecmd, shlex.quote(repourl), search)
+                (ud.basecmd, repourl, search)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, cmd, repourl)
             output = runfetchcmd(cmd, d, True)
@@ -732,11 +588,10 @@ class Git(FetchMethod):
         """
         pupver = ('', '')
 
-        tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX') or r"(?P<pver>([0-9][\.|_]?)+)")
+        tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX') or "(?P<pver>([0-9][\.|_]?)+)")
         try:
             output = self._lsremote(ud, d, "refs/tags/*")
-        except (bb.fetch2.FetchError, bb.fetch2.NetworkAccess) as e:
-            bb.note("Could not list remote: %s" % str(e))
+        except bb.fetch2.FetchError or bb.fetch2.NetworkAccess:
             return pupver
 
         verstring = ""
@@ -747,13 +602,13 @@ class Git(FetchMethod):
 
             tag_head = line.split("/")[-1]
             # Ignore non-released branches
-            m = re.search(r"(alpha|beta|rc|final)+", tag_head)
+            m = re.search("(alpha|beta|rc|final)+", tag_head)
             if m:
                 continue
 
             # search for version in the line
             tag = tagregex.search(tag_head)
-            if tag is None:
+            if tag == None:
                 continue
 
             tag = tag.group('pver')

bitbake/lib/bb/fetch2/gitannex.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 """
 BitBake 'Fetch' git annex implementation
 """
@@ -5,12 +7,24 @@ BitBake 'Fetch' git annex implementation
 # Copyright (C) 2014 Otavio Salvador
 # Copyright (C) 2014 O.S. Systems Software LTDA.
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
+import os
 import bb
 from   bb.fetch2.git import Git
 from   bb.fetch2 import runfetchcmd
+from   bb.fetch2 import logger
 
 class GitANNEX(Git):
     def supports(self, ud, d):

bitbake/lib/bb/fetch2/gitsm.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 """
 BitBake 'Fetch' git submodules implementation
 
@@ -14,18 +16,24 @@ NOTE: Switching a SRC_URI from "git://" to "gitsm://" requires a clean of your r
 
 # Copyright (C) 2013 Richard Purdie
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
 #
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import os
 import bb
-import copy
-import shutil
-import tempfile
 from   bb.fetch2.git import Git
 from   bb.fetch2 import runfetchcmd
 from   bb.fetch2 import logger
-from   bb.fetch2 import Fetch
 
 class GitSM(Git):
     def supports(self, ud, d):
@@ -34,192 +42,94 @@ class GitSM(Git):
         """
         return ud.type in ['gitsm']
 
-    def process_submodules(self, ud, workdir, function, d):
-        """
-        Iterate over all of the submodules in this repository and execute
-        the 'function' for each of them.
-        """
-
-        submodules = []
-        paths = {}
-        revision = {}
-        uris = {}
-        subrevision = {}
-
-        def parse_gitmodules(gitmodules):
-            modules = {}
-            module = ""
-            for line in gitmodules.splitlines():
-                if line.startswith('[submodule'):
-                    module = line.split('"')[1]
-                    modules[module] = {}
-                elif module and line.strip().startswith('path'):
-                    path = line.split('=')[1].strip()
-                    modules[module]['path'] = path
-                elif module and line.strip().startswith('url'):
-                    url = line.split('=')[1].strip()
-                    modules[module]['url'] = url
-            return modules
-
-        # Collect the defined submodules, and their attributes
+    def uses_submodules(self, ud, d, wd):
         for name in ud.names:
             try:
-                gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
-            except:
-                # No submodules to update
-                continue
-
-            for m, md in parse_gitmodules(gitmodules).items():
-                try:
-                    module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=True, workdir=workdir)
-                except:
-                    # If the command fails, we don't have a valid file to check.  If it doesn't
-                    # fail -- it still might be a failure, see next check...
-                    module_hash = ""
-
-                if not module_hash:
-                    logger.debug(1, "submodule %s is defined, but is not initialized in the repository. Skipping", m)
-                    continue
-
-                submodules.append(m)
-                paths[m] = md['path']
-                revision[m] = ud.revisions[name]
-                uris[m] = md['url']
-                subrevision[m] = module_hash.split()[2]
-
-                # Convert relative to absolute uri based on parent uri
-                if uris[m].startswith('..'):
-                    newud = copy.copy(ud)
-                    newud.path = os.path.realpath(os.path.join(newud.path, uris[m]))
-                    uris[m] = Git._get_repo_url(self, newud)
-
-        for module in submodules:
-            # Translate the module url into a SRC_URI
-
-            if "://" in uris[module]:
-                # Properly formated URL already
-                proto = uris[module].split(':', 1)[0]
-                url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
-            else:
-                if ":" in uris[module]:
-                    # Most likely an SSH style reference
-                    proto = "ssh"
-                    if ":/" in uris[module]:
-                        # Absolute reference, easy to convert..
-                        url = "gitsm://" + uris[module].replace(':/', '/', 1)
-                    else:
-                        # Relative reference, no way to know if this is right!
-                        logger.warning("Submodule included by %s refers to relative ssh reference %s.  References may fail if not absolute." % (ud.url, uris[module]))
-                        url = "gitsm://" + uris[module].replace(':', '/', 1)
-                else:
-                    # This has to be a file reference
-                    proto = "file"
-                    url = "gitsm://" + uris[module]
-
-            url += ';protocol=%s' % proto
-            url += ";name=%s" % module
-            url += ";subpath=%s" % module
-
-            ld = d.createCopy()
-            # Not necessary to set SRC_URI, since we're passing the URI to
-            # Fetch.
-            #ld.setVar('SRC_URI', url)
-            ld.setVar('SRCREV_%s' % module, subrevision[module])
-
-            # Workaround for issues with SRCPV/SRCREV_FORMAT errors
-            # error refer to 'multiple' repositories.  Only the repository
-            # in the original SRC_URI actually matters...
-            ld.setVar('SRCPV', d.getVar('SRCPV'))
-            ld.setVar('SRCREV_FORMAT', module)
-
-            function(ud, url, module, paths[module], workdir, ld)
-
-        return submodules != []
-
-    def need_update(self, ud, d):
-        if Git.need_update(self, ud, d):
-            return True
-
-        try:
-            # Check for the nugget dropped by the download operation
-            known_srcrevs = runfetchcmd("%s config --get-all bitbake.srcrev" % \
-                            (ud.basecmd), d, workdir=ud.clonedir)
-
-            if ud.revisions[ud.names[0]] not in known_srcrevs.split():
+                runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=wd)
                 return True
-        except bb.fetch2.FetchError:
-            # No srcrev nuggets, so this is new and needs to be updated
-            return True
-
+            except bb.fetch.FetchError:
+                pass
         return False
 
+    def _set_relative_paths(self, repopath):
+        """
+        Fix submodule paths to be relative instead of absolute,
+        so that when we move the repo it doesn't break
+        (In Git 1.7.10+ this is done automatically)
+        """
+        submodules = []
+        with open(os.path.join(repopath, '.gitmodules'), 'r') as f:
+            for line in f.readlines():
+                if line.startswith('[submodule'):
+                    submodules.append(line.split('"')[1])
+
+        for module in submodules:
+            repo_conf = os.path.join(repopath, module, '.git')
+            if os.path.exists(repo_conf):
+                with open(repo_conf, 'r') as f:
+                    lines = f.readlines()
+                newpath = ''
+                for i, line in enumerate(lines):
+                    if line.startswith('gitdir:'):
+                        oldpath = line.split(': ')[-1].rstrip()
+                        if oldpath.startswith('/'):
+                            newpath = '../' * (module.count('/') + 1) + '.git/modules/' + module
+                            lines[i] = 'gitdir: %s\n' % newpath
+                            break
+                if newpath:
+                    with open(repo_conf, 'w') as f:
+                        for line in lines:
+                            f.write(line)
+
+            repo_conf2 = os.path.join(repopath, '.git', 'modules', module, 'config')
+            if os.path.exists(repo_conf2):
+                with open(repo_conf2, 'r') as f:
+                    lines = f.readlines()
+                newpath = ''
+                for i, line in enumerate(lines):
+                    if line.lstrip().startswith('worktree = '):
+                        oldpath = line.split(' = ')[-1].rstrip()
+                        if oldpath.startswith('/'):
+                            newpath = '../' * (module.count('/') + 3) + module
+                            lines[i] = '\tworktree = %s\n' % newpath
+                            break
+                if newpath:
+                    with open(repo_conf2, 'w') as f:
+                        for line in lines:
+                            f.write(line)
+
+    def update_submodules(self, ud, d):
+        # We have to convert bare -> full repo, do the submodule bit, then convert back
+        tmpclonedir = ud.clonedir + ".tmp"
+        gitdir = tmpclonedir + os.sep + ".git"
+        bb.utils.remove(tmpclonedir, True)
+        os.mkdir(tmpclonedir)
+        os.rename(ud.clonedir, gitdir)
+        runfetchcmd("sed " + gitdir + "/config -i -e 's/bare.*=.*true/bare = false/'", d)
+        runfetchcmd(ud.basecmd + " reset --hard", d, workdir=tmpclonedir)
+        runfetchcmd(ud.basecmd + " checkout -f " + ud.revisions[ud.names[0]], d, workdir=tmpclonedir)
+        runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=tmpclonedir)
+        self._set_relative_paths(tmpclonedir)
+        runfetchcmd("sed " + gitdir + "/config -i -e 's/bare.*=.*false/bare = true/'", d, workdir=tmpclonedir)
+        os.rename(gitdir, ud.clonedir,)
+        bb.utils.remove(tmpclonedir, True)
+
     def download(self, ud, d):
-        def download_submodule(ud, url, module, modpath, workdir, d):
-            url += ";bareclone=1;nobranch=1"
-
-            # Is the following still needed?
-            #url += ";nocheckout=1"
-
-            try:
-                newfetch = Fetch([url], d, cache=False)
-                newfetch.download()
-                # Drop a nugget to add each of the srcrevs we've fetched (used by need_update)
-                runfetchcmd("%s config --add bitbake.srcrev %s" % \
-                            (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=workdir)
-            except Exception as e:
-                logger.error('gitsm: submodule download failed: %s %s' % (type(e).__name__, str(e)))
-                raise
-
         Git.download(self, ud, d)
 
-        # If we're using a shallow mirror tarball it needs to be unpacked
-        # temporarily so that we can examine the .gitmodules file
-        if ud.shallow and os.path.exists(ud.fullshallow) and self.need_update(ud, d):
-            tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR"))
-            runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=tmpdir)
-            self.process_submodules(ud, tmpdir, download_submodule, d)
-            shutil.rmtree(tmpdir)
-        else:
-            self.process_submodules(ud, ud.clonedir, download_submodule, d)
+        if not ud.shallow or ud.localpath != ud.fullshallow:
+            submodules = self.uses_submodules(ud, d, ud.clonedir)
+            if submodules:
+                self.update_submodules(ud, d)
+
+    def clone_shallow_local(self, ud, dest, d):
+        super(GitSM, self).clone_shallow_local(ud, dest, d)
+
+        runfetchcmd('cp -fpPRH "%s/modules" "%s/"' % (ud.clonedir, os.path.join(dest, '.git')), d)
 
     def unpack(self, ud, destdir, d):
-        def unpack_submodules(ud, url, module, modpath, workdir, d):
-            url += ";bareclone=1;nobranch=1"
-
-            # Figure out where we clone over the bare submodules...
-            if ud.bareclone:
-                repo_conf = ud.destdir
-            else:
-                repo_conf = os.path.join(ud.destdir, '.git')
-
-            try:
-                newfetch = Fetch([url], d, cache=False)
-                newfetch.unpack(root=os.path.dirname(os.path.join(repo_conf, 'modules', module)))
-            except Exception as e:
-                logger.error('gitsm: submodule unpack failed: %s %s' % (type(e).__name__, str(e)))
-                raise
-
-            local_path = newfetch.localpath(url)
-
-            # Correct the submodule references to the local download version...
-            runfetchcmd("%(basecmd)s config submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' : local_path}, d, workdir=ud.destdir)
-
-            if ud.shallow:
-                runfetchcmd("%(basecmd)s config submodule.%(module)s.shallow true" % {'basecmd': ud.basecmd, 'module': module}, d, workdir=ud.destdir)
-
-            # Ensure the submodule repository is NOT set to bare, since we're checking it out...
-            try:
-                runfetchcmd("%s config core.bare false" % (ud.basecmd), d, quiet=True, workdir=os.path.join(repo_conf, 'modules', module))
-            except:
-                logger.error("Unable to set git config core.bare to false for %s" % os.path.join(repo_conf, 'modules', module))
-                raise
-
         Git.unpack(self, ud, destdir, d)
 
-        ret = self.process_submodules(ud, ud.destdir, unpack_submodules, d)
-
-        if not ud.bareclone and ret:
-            # All submodules should already be downloaded and configured in the tree.  This simply sets
-            # up the configuration and checks out the files.  The main project config should remain
-            # unmodified, and no download from the internet should occur.
-            runfetchcmd("%s submodule update --recursive --no-fetch" % (ud.basecmd), d, quiet=True, workdir=ud.destdir)
+        if self.uses_submodules(ud, d, ud.destdir):
+            runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
+            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)

bitbake/lib/bb/fetch2/hg.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 """
 BitBake 'Fetch' implementation for mercurial DRCS (hg).
 
@@ -7,12 +9,24 @@ BitBake 'Fetch' implementation for mercurial DRCS (hg).
 # Copyright (C) 2004        Marcin Juszkiewicz
 # Copyright (C) 2007        Robert Schuster
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 #
 # Based on functions from the base bb module, Copyright 2003 Holger Schurig
-#
 
 import os
+import sys
+import logging
 import bb
 import errno
 from bb.fetch2 import FetchMethod
@@ -52,6 +66,13 @@ class Hg(FetchMethod):
         else:
             ud.proto = "hg"
 
+        ud.setup_revisions(d)
+
+        if 'rev' in ud.parm:
+            ud.revision = ud.parm['rev']
+        elif not ud.revision:
+            ud.revision = self.latest_revision(ud, d)
+
         # Create paths to mercurial checkouts
         hgsrcname = '%s_%s_%s' % (ud.module.replace('/', '.'), \
                             ud.host, ud.path.replace('/', '.'))
@@ -59,19 +80,12 @@ class Hg(FetchMethod):
         ud.fullmirror = os.path.join(d.getVar("DL_DIR"), mirrortarball)
         ud.mirrortarballs = [mirrortarball]
 
-        hgdir = d.getVar("HGDIR") or (d.getVar("DL_DIR") + "/hg")
+        hgdir = d.getVar("HGDIR") or (d.getVar("DL_DIR") + "/hg/")
         ud.pkgdir = os.path.join(hgdir, hgsrcname)
         ud.moddir = os.path.join(ud.pkgdir, ud.module)
         ud.localfile = ud.moddir
         ud.basecmd = d.getVar("FETCHCMD_hg") or "/usr/bin/env hg"
 
-        ud.setup_revisions(d)
-
-        if 'rev' in ud.parm:
-            ud.revision = ud.parm['rev']
-        elif not ud.revision:
-            ud.revision = self.latest_revision(ud, d)
-
         ud.write_tarballs = d.getVar("BB_GENERATE_MIRROR_TARBALLS")
 
     def need_update(self, ud, d):
@@ -85,7 +99,7 @@ class Hg(FetchMethod):
     def try_premirror(self, ud, d):
         # If we don't do this, updating an existing checkout with only premirrors
         # is not possible
-        if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")):
+        if d.getVar("BB_FETCH_PREMIRRORONLY") is not None:
             return True
         if os.path.exists(ud.moddir):
             return False
@@ -137,7 +151,7 @@ class Hg(FetchMethod):
                 cmd = "%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" pull" % (ud.basecmd, ud.user, ud.pswd, proto)
             else:
                 cmd = "%s pull" % (ud.basecmd)
-        elif command == "update" or command == "up":
+        elif command == "update":
             if ud.user and ud.pswd:
                 cmd = "%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" update -C %s" % (ud.basecmd, ud.user, ud.pswd, proto, " ".join(options))
             else:
@@ -245,19 +259,12 @@ class Hg(FetchMethod):
 
         scmdata = ud.parm.get("scmdata", "")
         if scmdata != "nokeep":
-            proto = ud.parm.get('protocol', 'http')
             if not os.access(os.path.join(codir, '.hg'), os.R_OK):
                 logger.debug(2, "Unpack: creating new hg repository in '" + codir + "'")
                 runfetchcmd("%s init %s" % (ud.basecmd, codir), d)
             logger.debug(2, "Unpack: updating source in '" + codir + "'")
-            if ud.user and ud.pswd:
-                runfetchcmd("%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" pull %s" % (ud.basecmd, ud.user, ud.pswd, proto, ud.moddir), d, workdir=codir)
-            else:
-                runfetchcmd("%s pull %s" % (ud.basecmd, ud.moddir), d, workdir=codir)
-            if ud.user and ud.pswd:
-                runfetchcmd("%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" up -C %s" % (ud.basecmd, ud.user, ud.pswd, proto, revflag), d, workdir=codir)
-            else:
-                runfetchcmd("%s up -C %s" % (ud.basecmd, revflag), d, workdir=codir)
+            runfetchcmd("%s pull %s" % (ud.basecmd, ud.moddir), d, workdir=codir)
+            runfetchcmd("%s up -C %s" % (ud.basecmd, revflag), d, workdir=codir)
         else:
             logger.debug(2, "Unpack: extracting source to '" + codir + "'")
             runfetchcmd("%s archive -t files %s %s" % (ud.basecmd, revflag, codir), d, workdir=ud.moddir)

bitbake/lib/bb/fetch2/local.py

@@ -1,3 +1,5 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
 """
 BitBake 'Fetch' implementations
 
@@ -8,10 +10,20 @@ BitBake build tools.
 
 # Copyright (C) 2003, 2004  Chris Larson
 #
-# SPDX-License-Identifier: GPL-2.0-only
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 #
 # Based on functions from the base bb module, Copyright 2003 Holger Schurig
-#
 
 import os
 import urllib.request, urllib.parse, urllib.error