build-appliance-image: Update to krogoth head revision

(From OE-Core rev: 1dc9ce406497d6e996a40afc53293d9a576c8314)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

.gitignore

@@ -1,14 +1,12 @@
 *.pyc
 *.pyo
 /*.patch
-/.repo/
 /build*/
 pyshtables.py
 pstage/
 scripts/oe-git-proxy-socks
 sources/
 meta-*/
-buildtools/
 !meta-skeleton
 !meta-selftest
 hob-image-*.bb
@@ -20,20 +18,10 @@ hob-image-*.bb
 !meta-yocto
 !meta-yocto-bsp
 !meta-yocto-imported
-/documentation/*/eclipse/
-/documentation/*/*.html
-/documentation/*/*.pdf
-/documentation/*/*.tgz
-/bitbake/doc/bitbake-user-manual/bitbake-user-manual.html
-/bitbake/doc/bitbake-user-manual/bitbake-user-manual.pdf
-/bitbake/doc/bitbake-user-manual/bitbake-user-manual.tgz
+documentation/user-manual/user-manual.html
+documentation/user-manual/user-manual.pdf
+documentation/user-manual/user-manual.tgz
 pull-*/
 bitbake/lib/toaster/contrib/tts/backlog.txt
 bitbake/lib/toaster/contrib/tts/log/*
-bitbake/lib/toaster/contrib/tts/.cache/*
-bitbake/lib/bb/tests/runqueue-tests/bitbake-cookerdaemon.log
-_toaster_clones/
-downloads/
-sstate-cache/
-toaster.sqlite
-.vscode/
+bitbake/lib/toaster/contrib/tts/.cache/*

.templateconf

@@ -0,0 +1,2 @@
+# Template settings
+TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}

LICENSE

@@ -0,0 +1,14 @@
+Different components of OpenEmbedded are under different licenses (a mix
+of MIT and GPLv2). Please see:
+
+meta/COPYING.GPLv2 (GPLv2)
+meta/COPYING.MIT (MIT)
+meta-selftest/COPYING.MIT (MIT)
+meta-skeleton/COPYING.MIT (MIT)
+
+All metadata is MIT licensed unless otherwise stated. Source code
+included in tree for individual recipes is under the LICENSE stated in
+the associated recipe (.bb file) unless otherwise stated.
+
+License information for any other files is either explicitly stated 
+or defaults to GPL version 2.

README

@@ -1,114 +1,58 @@
-The poky repository master branch is no longer being updated.
+Poky
+====
 
-You can either:
+Poky is an integration of various components to form a complete prepackaged
+build system and development environment. It features support for building
+customised embedded device style images. There are reference demo images
+featuring a X11/Matchbox/GTK themed UI called Sato. The system supports
+cross-architecture application development using QEMU emulation and a
+standalone toolchain and SDK with IDE integration.
 
-a) switch to individual clones of bitbake, openembedded-core, meta-yocto and yocto-docs
+Additional information on the specifics of hardware that Poky supports
+is available in README.hardware. Further hardware support can easily be added
+in the form of layers which extend the systems capabilities in a modular way.
 
-   https://docs.yoctoproject.org/dev/dev-manual/poky-manual-setup.html
+As an integration layer Poky consists of several upstream projects such as 
+BitBake, OpenEmbedded-Core, Yocto documentation and various sources of information 
+e.g. for the hardware support. Poky is in turn a component of the Yocto Project.
 
-b) use the new bitbake-setup
+The Yocto Project has extensive documentation about the system including a 
+reference manual which can be found at:
+    http://yoctoproject.org/documentation
 
-     https://docs.yoctoproject.org/bitbake/dev/bitbake-user-manual/bitbake-user-manual-environment-setup.html
+OpenEmbedded-Core is a layer containing the core metadata for current versions
+of OpenEmbedded. It is distro-less (can build a functional image with
+DISTRO = "nodistro") and contains only emulated machine support.
 
-You can find more information in our documentation: https://docs.yoctoproject.org/
+For information about OpenEmbedded, see the OpenEmbedded website:
+    http://www.openembedded.org/
 
-Note that "poky" the distro setting is still available in meta-yocto as
-before and we continue to use and maintain that.
+Where to Send Patches
+=====================
 
-Long live Poky!
+As Poky is an integration repository (built using a tool called combo-layer),
+patches against the various components should be sent to their respective
+upstreams:
 
+bitbake:
+    Git repository: http://git.openembedded.org/bitbake/
+    Mailing list: bitbake-devel@lists.openembedded.org
 
+documentation:
+    Git repository: http://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/
+    Mailing list: yocto@yoctoproject.org
 
+meta-poky, meta-yocto-bsp:
+    Git repository: http://git.yoctoproject.org/cgit/cgit.cgi/meta-yocto(-bsp)
+    Mailing list: poky@yoctoproject.org
 
-Some further information on the background of this change follows. The 
-details are taken from:
-https://lists.openembedded.org/g/openembedded-architecture/message/2179
+Everything else should be sent to the OpenEmbedded Core mailing list.  If in
+doubt, check the oe-core git repository for the content you intend to modify.
+Before sending, be sure the patches apply cleanly to the current oe-core git
+repository.
 
-TLDR: People have complained about the combo-layer built poky
-repository for years. It was meant to be a temporary thing, we now have
-an alternative and I'm therefore doing what I promised I'd do. Change
-is tough, things may break but this is the right point to at least try
-it.
-
-I'd like to note that:
-  * setting up builds with a separate oe-core and bitbake clone 
-    works as it always has done
-  * you can change your CI just to use those two repos instead of poky
-  * bitbake-setup isn't mandatory, it will just be what the yocto-
-    docs presents to users
-  * we don't have to stop maintaining the poky repository 
-    however nobody will test the new approach/code unless we do
-  * we are optionally exposing sstate mirrors in the new config
-  * we are also exposing config fragments to users
-  * poky as a DISTRO in meta-yocto remains
-
-A bit more about the history and background for those who are
-interested and then some FAQs:
-
-Back around 2010 when we split up openembedded-classic and started
-developing layers, we made the artificial "poky" repository construct
-as a way to let people easily and quickly get started with the project.
-without cloning and managing multiple repositories. Layers were a new
-idea with lots of rough edges. kas didn't exist, I think repo was only
-just created and it was a different world. For us, it meant hacking up
-a quick tool, "combo-layer" and it was really a temporary solution to
-fill a gap and it was at least as functional as repo of the era. It was
-assumed we'd work it out properly in the future.
-
-At developer meetings there are inevitable questions about why
-poky/combo-layer exist and few seem to actually like/support it. There
-are continual questions about why a tool doesn't exist or why we don't
-adopt one too.
-
-15 years later, a bit longer than we might have thought, we are finally
-in a position where there may be a viable way forward to change.
-
-It has taken us a bit of time to get to this point. I wrote the
-original description of something like bitbake-setup about 7-8 years
-ago. I shared it privately with a few people, the review feedback
-stopped me pushing it further as I simply did not have the bandwidth.
-We were fortunate to get funding from the Sovereign Tech Fund to start
-the work and whilst I'd probably prefer to avoid the issue, the time
-had come to start. Since then, Alexander Kanavin has put a lot of work
-into getting it to the point where it would be possible to switch. A
-huge thanks to him for getting this to the current point.
-
-Why not use kas/submodules/repo?
-
-This topic has been discussed in depth several times. Very roughly,
-these are either difficult to focus on our use cases or have specific
-designs and intent which we as a project would struggle to influence.
-We are taking significant influence from some of them but also trying
-to build something where we can benefit from tight direct integration
-with bitbake and the metadata. For example fragment support is generic
-and hopefully something other approaches can also benefit from. We want
-to provide something we can switch the projects docs and autobuilder to
-which we can control and develop as we need it to. We are not aiming to
-force anyone to switch, you can use whichever tool you want.
-
-Can we not keep poky [repository master branch] around?
-
-If we do that, nobody will use the new tooling and it will be a
-disaster as issues won't get resolved. We need our CI to use the same
-thing we promote to our new and experienced users. We need this new
-tooling to be usable by our experienced developers too. We have tried
-for months to get people to try it and they simply don't. Making a
-release with it won't change much either. It needs people using it and
-for that, poky has to stop being updated.
-
-What happens to poky [repository]?
-
-The LTS branches continue their lifetime as planned. For master, I'll
-probably put a final commit in changing to just a README which points
-people at the bitbake-setup changes and explains what happened.
-
-What are the timelines? Why now?
-
-If we're going to make a change, we really want this in the next LTS
-release, which is April 2026. We only have one release before that
-which is now, October 2025. We therefore need to switch now, and then
-give us time to update docs, fix issues that arise and so on and have
-it in a release cycle. Whilst it means delaying the Oct 2025 release
-slightly, that is the right thing to do in the context of the bigger
-picture.
+    Git repository: http://git.openembedded.org/openembedded-core/
+    Mailing list: openembedded-core@lists.openembedded.org
 
+Note: The scripts directory should be treated with extra care as it is a mix of
+oe-core and poky-specific files.

README.hardware

@@ -0,0 +1,500 @@
+                          Poky Hardware README
+                          ====================
+
+This file gives details about using Poky with the reference machines
+supported out of the box. A full list of supported reference target machines
+can be found by looking in the following directories:
+
+   meta/conf/machine/
+   meta-yocto-bsp/conf/machine/
+
+If you are in doubt about using Poky/OpenEmbedded with your hardware, consult
+the documentation for your board/device.
+
+Support for additional devices is normally added by creating BSP layers - for
+more information please see the Yocto Board Support Package (BSP) Developer's
+Guide - documentation source is in documentation/bspguide or download the PDF
+from:
+
+   http://yoctoproject.org/documentation
+
+Support for physical reference hardware has now been split out into a
+meta-yocto-bsp layer which can be removed separately from other layers if not
+needed.
+
+
+QEMU Emulation Targets
+======================
+
+To simplify development, the build system supports building images to
+work with the QEMU emulator in system emulation mode. Several architectures
+are currently supported:
+
+  * ARM (qemuarm)
+  * x86 (qemux86)
+  * x86-64 (qemux86-64)
+  * PowerPC (qemuppc)
+  * MIPS (qemumips)
+
+Use of the QEMU images is covered in the Yocto Project Reference Manual.
+The appropriate MACHINE variable value corresponding to the target is given
+in brackets.
+
+
+Hardware Reference Boards
+=========================
+
+The following boards are supported by the meta-yocto-bsp layer:
+
+  * Texas Instruments Beaglebone (beaglebone)
+  * Freescale MPC8315E-RDB (mpc8315e-rdb)
+
+For more information see the board's section below. The appropriate MACHINE
+variable value corresponding to the board is given in brackets.
+
+Reference Board Maintenance
+===========================
+
+Send pull requests, patches, comments or questions about meta-yocto-bsps to poky@yoctoproject.org
+
+Maintainers: Kevin Hao <kexin.hao@windriver.com>
+             Bruce Ashfield <bruce.ashfield@windriver.com>
+
+Consumer Devices
+================
+
+The following consumer devices are supported by the meta-yocto-bsp layer:
+
+  * Intel x86 based PCs and devices (genericx86)
+  * Ubiquiti Networks EdgeRouter Lite (edgerouter)
+
+For more information see the device's section below. The appropriate MACHINE
+variable value corresponding to the device is given in brackets.
+
+
+
+                      Specific Hardware Documentation
+                      ===============================
+
+
+Intel x86 based PCs and devices (genericx86)
+==========================================
+
+The genericx86 MACHINE is tested on the following platforms:
+
+Intel Xeon/Core i-Series:
+  + Intel Romley Server: Sandy Bridge Xeon processor, C600 PCH (Patsburg), (Canoe Pass CRB)
+  + Intel Romley Server: Ivy Bridge Xeon processor, C600 PCH (Patsburg), (Intel SDP S2R3)
+  + Intel Crystal Forest Server: Sandy Bridge Xeon processor, DH89xx PCH (Cave Creek), (Stargo CRB)
+  + Intel Chief River Mobile: Ivy Bridge Mobile processor, QM77 PCH (Panther Point-M), (Emerald Lake II CRB, Sabino Canyon CRB)
+  + Intel Huron River Mobile: Sandy Bridge processor, QM67 PCH (Cougar Point), (Emerald Lake CRB, EVOC EC7-1817LNAR board)
+  + Intel Calpella Platform: Core i7 processor, QM57 PCH (Ibex Peak-M), (Red Fort CRB, Emerson MATXM CORE-411-B)
+  + Intel Nehalem/Westmere-EP Server: Xeon 56xx/55xx processors, 5520 chipset, ICH10R IOH (82801), (Hanlan Creek CRB)
+  + Intel Nehalem Workstation: Xeon 56xx/55xx processors, System SC5650SCWS (Greencity CRB)
+  + Intel Picket Post Server: Xeon 56xx/55xx processors (Jasper Forest), 3420 chipset (Ibex Peak), (Osage CRB)
+  + Intel Storage Platform: Sandy Bridge Xeon processor, C600 PCH (Patsburg), (Oak Creek Canyon CRB)
+  + Intel Shark Bay Client Platform: Haswell processor, LynxPoint PCH, (Walnut Canyon CRB, Lava Canyon CRB, Basking Ridge CRB, Flathead Creek CRB)
+  + Intel Shark Bay Ultrabook Platform: Haswell ULT processor, Lynx Point-LP PCH, (WhiteTip Mountain 1 CRB)
+
+Intel Atom platforms:
+  + Intel embedded Menlow: Intel Atom Z510/530 CPU, System Controller Hub US15W (Portwell NANO-8044)
+  + Intel Luna Pier: Intel Atom N4xx/D5xx series CPU (aka: Pineview-D & -M), 82801HM I/O Hub (ICH8M), (Advantech AIMB-212, Moon Creek CRB)
+  + Intel Queens Bay platform: Intel Atom E6xx CPU (aka: Tunnel Creek), Topcliff EG20T I/O Hub (Emerson NITX-315, Crown Bay CRB, Minnow Board)
+  + Intel Fish River Island platform: Intel Atom E6xx CPU (aka: Tunnel Creek), Topcliff EG20T I/O Hub (Kontron KM2M806)
+  + Intel Cedar Trail platform: Intel Atom N2000 & D2000 series CPU (aka: Cedarview), NM10 Express Chipset (Norco kit BIS-6630, Cedar Rock CRB)
+
+and is likely to work on many unlisted Atom/Core/Xeon based devices. The MACHINE
+type supports ethernet, wifi, sound, and Intel/vesa graphics by default in
+addition to common PC input devices, busses, and so on.
+
+Depending on the device, it can boot from a traditional hard-disk, a USB device,
+or over the network. Writing generated images to physical media is
+straightforward with a caveat for USB devices. The following examples assume the
+target boot device is /dev/sdb, be sure to verify this and use the correct
+device as the following commands are run as root and are not reversable.
+
+USB Device:
+  1. Build a live image. This image type consists of a simple filesystem
+     without a partition table, which is suitable for USB keys, and with the
+     default setup for the genericx86 machine, this image type is built
+     automatically for any image you build. For example:
+
+     $ bitbake core-image-minimal
+
+  2. Use the "dd" utility to write the image to the raw block device. For
+     example:
+
+     # dd if=core-image-minimal-genericx86.hddimg of=/dev/sdb
+
+  If the device fails to boot with "Boot error" displayed, or apparently
+  stops just after the SYSLINUX version banner, it is likely the BIOS cannot
+  understand the physical layout of the disk (or rather it expects a
+  particular layout and cannot handle anything else). There are two possible
+  solutions to this problem:
+
+  1. Change the BIOS USB Device setting to HDD mode. The label will vary by
+     device, but the idea is to force BIOS to read the Cylinder/Head/Sector
+     geometry from the device.
+
+  2. Without such an option, the BIOS generally boots the device in USB-ZIP
+     mode. To write an image to a USB device that will be bootable in
+     USB-ZIP mode, carry out the following actions:
+
+     a. Determine the geometry of your USB device using fdisk:
+
+     # fdisk /dev/sdb
+     Command (m for help): p
+
+     Disk /dev/sdb: 4011 MB, 4011491328 bytes
+     124 heads, 62 sectors/track, 1019 cylinders, total 7834944 sectors
+     ...
+
+     Command (m for help): q
+
+     b. Configure the USB device for USB-ZIP mode:
+     
+     # mkdiskimage -4 /dev/sdb 1019 124 62
+
+     Where 1019, 124 and 62 are the cylinder, head and sectors/track counts
+     as reported by fdisk (substitute the values reported for your device).
+     When the operation has finished and the access LED (if any) on the
+     device stops flashing, remove and reinsert the device to allow the
+     kernel to detect the new partition layout.
+
+     c. Copy the contents of the image to the USB-ZIP mode device:
+
+     # mkdir /tmp/image
+     # mkdir /tmp/usbkey
+     # mount -o loop core-image-minimal-genericx86.hddimg  /tmp/image
+     # mount /dev/sdb4 /tmp/usbkey
+     # cp -rf /tmp/image/* /tmp/usbkey
+
+     d. Install the syslinux boot loader:
+
+     # syslinux /dev/sdb4
+
+     e. Unmount everything:
+
+     # umount /tmp/image
+     # umount /tmp/usbkey
+
+  Install the boot device in the target board and configure the BIOS to boot
+  from it.
+
+  For more details on the USB-ZIP scenario, see the syslinux documentation:
+  http://git.kernel.org/?p=boot/syslinux/syslinux.git;a=blob_plain;f=doc/usbkey.txt;hb=HEAD
+
+
+Texas Instruments Beaglebone (beaglebone)
+=========================================
+
+The Beaglebone is an ARM Cortex-A8 development board with USB, Ethernet, 2D/3D
+accelerated graphics, audio, serial, JTAG, and SD/MMC. The Black adds a faster
+CPU, more RAM, eMMC flash and a micro HDMI port. The beaglebone MACHINE is
+tested on the following platforms:
+
+  o Beaglebone Black A6
+  o Beaglebone A6 (the original "White" model)
+
+The Beaglebone Black has eMMC, while the White does not. Pressing the USER/BOOT
+button when powering on will temporarily change the boot order. But for the sake
+of simplicity, these instructions assume you have erased the eMMC on the Black,
+so its boot behavior matches that of the White and boots off of SD card. To do
+this, issue the following commands from the u-boot prompt:
+
+    # mmc dev 1
+    # mmc erase 0 512
+
+To further tailor these instructions for your board, please refer to the
+documentation at http://www.beagleboard.org/bone and http://www.beagleboard.org/black
+
+From a Linux system with access to the image files perform the following steps
+as root, replacing mmcblk0* with the SD card device on your machine (such as sdc
+if used via a usb card reader):
+
+  1. Partition and format an SD card:
+     # fdisk -lu /dev/mmcblk0
+
+     Disk /dev/mmcblk0: 3951 MB, 3951034368 bytes
+     255 heads, 63 sectors/track, 480 cylinders, total 7716864 sectors
+     Units = sectors of 1 * 512 = 512 bytes
+
+             Device Boot      Start         End      Blocks  Id System
+     /dev/mmcblk0p1   *          63      144584       72261   c Win95 FAT32 (LBA)
+     /dev/mmcblk0p2          144585      465884      160650  83 Linux
+
+     # mkfs.vfat -F 16 -n "boot" /dev/mmcblk0p1
+     # mke2fs -j -L "root" /dev/mmcblk0p2
+
+  The following assumes the SD card partitions 1 and 2 are mounted at
+  /media/boot and /media/root respectively. Removing the card and reinserting
+  it will do just that on most modern Linux desktop environments.
+
+  The files referenced below are made available after the build in
+  build/tmp/deploy/images.
+
+  2. Install the boot loaders
+     # cp MLO-beaglebone /media/boot/MLO
+     # cp u-boot-beaglebone.img /media/boot/u-boot.img
+
+  3. Install the root filesystem
+     # tar x -C /media/root -f core-image-$IMAGE_TYPE-beaglebone.tar.bz2
+
+  4. If using core-image-base or core-image-sato images, the SD card is ready
+     and rootfs already contains the kernel, modules and device tree (DTB)
+     files necessary to be booted with U-boot's default configuration, so
+     skip directly to step 8.
+     For core-image-minimal, proceed through next steps.
+
+  5. If using core-image-minimal rootfs, install the modules
+     # tar x -C /media/root -f modules-beaglebone.tgz
+
+  6. If using core-image-minimal rootfs, install the kernel zImage into /boot
+     directory of rootfs
+     # cp zImage-beaglebone.bin /media/root/boot/zImage
+
+  7. If using core-image-minimal rootfs, also install device tree (DTB) files
+     into /boot directory of rootfs
+     # cp zImage-am335x-bone.dtb /media/root/boot/am335x-bone.dtb
+     # cp zImage-am335x-boneblack.dtb /media/root/boot/am335x-boneblack.dtb
+
+  8. Unmount the SD partitions, insert the SD card into the Beaglebone, and
+     boot the Beaglebone
+
+
+Freescale MPC8315E-RDB (mpc8315e-rdb)
+=====================================
+
+The MPC8315 PowerPC reference platform (MPC8315E-RDB) is aimed at hardware and
+software development of network attached storage (NAS) and digital media server
+applications. The MPC8315E-RDB features the PowerQUICC II Pro processor, which
+includes a built-in security accelerator.
+
+(Note: you may find it easier to order MPC8315E-RDBA; this appears to be the
+same board in an enclosure with accessories. In any case it is fully
+compatible with the instructions given here.)
+
+Setup instructions
+------------------
+
+You will need the following:
+* NFS root setup on your workstation
+* TFTP server installed on your workstation
+* Straight-thru 9-conductor serial cable (DB9, M/F) connected from your 
+  PC to UART1
+* Ethernet connected to the first ethernet port on the board
+
+--- Preparation ---
+
+Note: if you have altered your board's ethernet MAC address(es) from the
+defaults, or you need to do so because you want multiple boards on the same
+network, then you will need to change the values in the dts file (patch
+linux/arch/powerpc/boot/dts/mpc8315erdb.dts within the kernel source). If
+you have left them at the factory default then you shouldn't need to do
+anything here.
+
+--- Booting from NFS root ---
+
+Load the kernel and dtb (device tree blob), and boot the system as follows:
+
+ 1. Get the kernel (uImage-mpc8315e-rdb.bin) and dtb (uImage-mpc8315e-rdb.dtb)
+    files from the tmp/deploy directory, and make them available on your TFTP
+    server.
+
+ 2. Connect the board's first serial port to your workstation and then start up
+    your favourite serial terminal so that you will be able to interact with
+    the serial console. If you don't have a favourite, picocom is suggested:
+
+  $ picocom /dev/ttyUSB0 -b 115200
+
+ 3. Power up or reset the board and press a key on the terminal when prompted
+    to get to the U-Boot command line
+
+ 4. Set up the environment in U-Boot:
+
+ => setenv ipaddr <board ip>
+ => setenv serverip <tftp server ip>
+ => setenv bootargs root=/dev/nfs rw nfsroot=<nfsroot ip>:<rootfs path> ip=<board ip>:<server ip>:<gateway ip>:255.255.255.0:mpc8315e:eth0:off console=ttyS0,115200
+
+ 5. Download the kernel and dtb, and boot:
+
+ => tftp 1000000 uImage-mpc8315e-rdb.bin
+ => tftp 2000000 uImage-mpc8315e-rdb.dtb
+ => bootm 1000000 - 2000000
+
+--- Booting from JFFS2 root ---
+
+ 1. First boot the board with NFS root.
+
+ 2. Erase the MTD partition which will be used as root:
+
+    $ flash_eraseall  /dev/mtd3
+
+ 3. Copy the JFFS2 image to the MTD partition:
+
+    $ flashcp core-image-minimal-mpc8315e-rdb.jffs2 /dev/mtd3
+
+ 4. Then reboot the board and set up the environment in U-Boot:
+
+    => setenv bootargs root=/dev/mtdblock3 rootfstype=jffs2 console=ttyS0,115200
+
+
+Ubiquiti Networks EdgeRouter Lite (edgerouter)
+==============================================
+
+The EdgeRouter Lite is part of the EdgeMax series. It is a MIPS64 router
+(based on the Cavium Octeon processor) with 512MB of RAM, which uses an
+internal USB pendrive for storage.
+
+Setup instructions
+------------------
+
+You will need the following:
+* RJ45 -> serial ("rollover") cable connected from your PC to the CONSOLE
+  port on the device
+* Ethernet connected to the first ethernet port on the board
+
+If using NFS as part of the setup process, you will also need:
+* NFS root setup on your workstation
+* TFTP server installed on your workstation (if fetching the kernel from
+  TFTP, see below).
+
+--- Preparation ---
+
+Build an image (e.g. core-image-minimal) using "edgerouter" as the MACHINE.
+In the following instruction it is based on core-image-minimal. Another target
+may be similiar with it.
+
+--- Booting from NFS root / kernel via TFTP ---
+
+Load the kernel, and boot the system as follows:
+
+ 1. Get the kernel (vmlinux) file from the tmp/deploy/images/edgerouter
+    directory, and make them available on your TFTP server.
+
+ 2. Connect the board's first serial port to your workstation and then start up
+    your favourite serial terminal so that you will be able to interact with
+    the serial console. If you don't have a favourite, picocom is suggested:
+
+  $ picocom /dev/ttyS0 -b 115200
+
+ 3. Power up or reset the board and press a key on the terminal when prompted
+    to get to the U-Boot command line
+
+ 4. Set up the environment in U-Boot:
+
+ => setenv ipaddr <board ip>
+ => setenv serverip <tftp server ip>
+
+ 5. Download the kernel and boot:
+
+ => tftp tftp $loadaddr vmlinux
+ => bootoctlinux $loadaddr coremask=0x3 root=/dev/nfs rw nfsroot=<nfsroot ip>:<rootfs path> ip=<board ip>:<server ip>:<gateway ip>:<netmask>:edgerouter:eth0:off mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@3072k(eeprom)
+
+--- Booting from USB root ---
+
+To boot from the USB disk, you either need to remove it from the edgerouter
+box and populate it from another computer, or use a previously booted NFS
+image and populate from the edgerouter itself.
+
+Type 1: Mounted USB disk
+------------------------
+
+To boot from the USB disk there are two available partitions on the factory
+USB storage. The rest of this guide assumes that these partitions are left
+intact. If you change the partition scheme, you must update your boot method
+appropriately.
+
+The standard partitions are:
+
+  - 1: vfat partition containing factory kernels
+  - 2: ext3 partition for the root filesystem.
+
+You can place the kernel on either partition 1, or partition 2, but the roofs
+must go on partition 2 (due to its size).
+
+Note: If you place the kernel on the ext3 partition, you must re-create the
+      ext3 filesystem, since the factory u-boot can only handle 128 byte inodes and
+      cannot read the partition otherwise.
+
+Steps:
+
+ 1. Remove the USB disk from the edgerouter and insert it into a computer
+    that has access to your build artifacts.
+
+ 2. Copy the kernel image to the USB storage (assuming discovered as 'sdb' on
+    the development machine):
+
+    2a) if booting from vfat
+ 
+        # mount /dev/sdb1 /mnt
+        # cp tmp/deploy/images/edgerouter/vmlinux /mnt
+        # umount /mnt
+
+    2b) if booting from ext3
+
+        # mkfs.ext3 -I 128 /dev/sdb2
+        # mount /dev/sdb2 /mnt
+        # mkdir /mnt/boot
+        # cp tmp/deploy/images/edgerouter/vmlinux /mnt/boot
+        # umount /mnt
+
+ 3. Extract the rootfs to the USB storage ext3 partition
+
+        # mount /dev/sdb2 /mnt
+        # tar -xvjpf core-image-minimal-XXX.tar.bz2 -C /mnt
+        # umount /mnt
+
+ 4. Reboot the board and press a key on the terminal when prompted to get to the U-Boot
+    command line:
+
+ 5. Load the kernel and boot:
+
+    5a) vfat boot
+
+         => fatload usb 0:1 $loadaddr vmlinux
+
+    5b) ext3 boot
+
+         => ext2load usb 0:2 $loadaddr boot/vmlinux
+ 
+    => bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rw rootwait mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@3072k(eeprom)
+ 
+
+Type 2: NFS
+-----------
+
+Note: If you place the kernel on the ext3 partition, you must re-create the
+      ext3 filesystem, since the factory u-boot can only handle 128 byte inodes and
+      cannot read the partition otherwise.
+
+      These boot instructions assume that you have recreated the ext3 filesystem with
+      128 byte inodes, you have an updated uboot or you are running and image capable
+      of making the filesystem on the board itself.
+
+
+ 1. Boot from NFS root
+
+ 2. Mount the USB disk partition 2 and then extract the contents of
+    tmp/deploy/core-image-XXXX.tar.bz2 into it.
+
+    Before starting, copy core-image-minimal-xxx.tar.bz2 and vmlinux into
+    rootfs path on your workstation.
+
+    and then,
+  
+      # mount /dev/sda2 /media/sda2
+      # tar -xvjpf core-image-minimal-XXX.tar.bz2 -C /media/sda2
+      # cp vmlinux /media/sda2/boot/vmlinux
+      # umount /media/sda2
+      # reboot
+
+ 3. Reboot the board and press a key on the terminal when prompted to get to the U-Boot
+    command line:
+
+    # reboot
+
+ 4. Load the kernel and boot:
+
+      => ext2load usb 0:2 $loadaddr boot/vmlinux
+      => bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rw rootwait mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@3072k(eeprom)

bitbake/AUTHORS

@@ -0,0 +1,10 @@
+Tim Ansell <mithro@mithis.net>
+Phil Blundell <pb@handhelds.org>
+Seb Frankengul <seb@frankengul.org>
+Holger Freyther <holger@moiji-mobile.com>
+Marcin Juszkiewicz <marcin@juszkiewicz.com.pl>
+Chris Larson <kergoth@handhelds.org>
+Ulrich Luckas <luckas@musoft.de>
+Mickey Lauer <mickey@Vanille.de>
+Richard Purdie <rpurdie@rpsys.net>
+Holger Schurig <holgerschurig@gmx.de>

bitbake/COPYING

@@ -0,0 +1,339 @@
+		    GNU GENERAL PUBLIC LICENSE
+		       Version 2, June 1991
+
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
+ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+			    Preamble
+
+  The licenses for most software are designed to take away your
+freedom to share and change it.  By contrast, the GNU General Public
+License is intended to guarantee your freedom to share and change free
+software--to make sure the software is free for all its users.  This
+General Public License applies to most of the Free Software
+Foundation's software and to any other program whose authors commit to
+using it.  (Some other Free Software Foundation software is covered by
+the GNU Lesser General Public License instead.)  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get it
+if you want it, that you can change the software or use pieces of it
+in new free programs; and that you know you can do these things.
+
+  To protect your rights, we need to make restrictions that forbid
+anyone to deny you these rights or to ask you to surrender the rights.
+These restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must give the recipients all the rights that
+you have.  You must make sure that they, too, receive or can get the
+source code.  And you must show them these terms so they know their
+rights.
+
+  We protect your rights with two steps: (1) copyright the software, and
+(2) offer you this license which gives you legal permission to copy,
+distribute and/or modify the software.
+
+  Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software.  If the software is modified by someone else and passed on, we
+want its recipients to know that what they have is not the original, so
+that any problems introduced by others will not reflect on the original
+authors' reputations.
+
+  Finally, any free program is threatened constantly by software
+patents.  We wish to avoid the danger that redistributors of a free
+program will individually obtain patent licenses, in effect making the
+program proprietary.  To prevent this, we have made it clear that any
+patent must be licensed for everyone's free use or not licensed at all.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+		    GNU GENERAL PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. This License applies to any program or other work which contains
+a notice placed by the copyright holder saying it may be distributed
+under the terms of this General Public License.  The "Program", below,
+refers to any such program or work, and a "work based on the Program"
+means either the Program or any derivative work under copyright law:
+that is to say, a work containing the Program or a portion of it,
+either verbatim or with modifications and/or translated into another
+language.  (Hereinafter, translation is included without limitation in
+the term "modification".)  Each licensee is addressed as "you".
+
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope.  The act of
+running the Program is not restricted, and the output from the Program
+is covered only if its contents constitute a work based on the
+Program (independent of having been made by running the Program).
+Whether that is true depends on what the Program does.
+
+  1. You may copy and distribute verbatim copies of the Program's
+source code as you receive it, in any medium, provided that you
+conspicuously and appropriately publish on each copy an appropriate
+copyright notice and disclaimer of warranty; keep intact all the
+notices that refer to this License and to the absence of any warranty;
+and give any other recipients of the Program a copy of this License
+along with the Program.
+
+You may charge a fee for the physical act of transferring a copy, and
+you may at your option offer warranty protection in exchange for a fee.
+
+  2. You may modify your copy or copies of the Program or any portion
+of it, thus forming a work based on the Program, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+    a) You must cause the modified files to carry prominent notices
+    stating that you changed the files and the date of any change.
+
+    b) You must cause any work that you distribute or publish, that in
+    whole or in part contains or is derived from the Program or any
+    part thereof, to be licensed as a whole at no charge to all third
+    parties under the terms of this License.
+
+    c) If the modified program normally reads commands interactively
+    when run, you must cause it, when started running for such
+    interactive use in the most ordinary way, to print or display an
+    announcement including an appropriate copyright notice and a
+    notice that there is no warranty (or else, saying that you provide
+    a warranty) and that users may redistribute the program under
+    these conditions, and telling the user how to view a copy of this
+    License.  (Exception: if the Program itself is interactive but
+    does not normally print such an announcement, your work based on
+    the Program is not required to print an announcement.)
+
+These requirements apply to the modified work as a whole.  If
+identifiable sections of that work are not derived from the Program,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works.  But when you
+distribute the same sections as part of a whole which is a work based
+on the Program, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+  3. You may copy and distribute the Program (or a work based on it,
+under Section 2) in object code or executable form under the terms of
+Sections 1 and 2 above provided that you also do one of the following:
+
+    a) Accompany it with the complete corresponding machine-readable
+    source code, which must be distributed under the terms of Sections
+    1 and 2 above on a medium customarily used for software interchange; or,
+
+    b) Accompany it with a written offer, valid for at least three
+    years, to give any third party, for a charge no more than your
+    cost of physically performing source distribution, a complete
+    machine-readable copy of the corresponding source code, to be
+    distributed under the terms of Sections 1 and 2 above on a medium
+    customarily used for software interchange; or,
+
+    c) Accompany it with the information you received as to the offer
+    to distribute corresponding source code.  (This alternative is
+    allowed only for noncommercial distribution and only if you
+    received the program in object code or executable form with such
+    an offer, in accord with Subsection b above.)
+
+The source code for a work means the preferred form of the work for
+making modifications to it.  For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to
+control compilation and installation of the executable.  However, as a
+special exception, the source code distributed need not include
+anything that is normally distributed (in either source or binary
+form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component
+itself accompanies the executable.
+
+If distribution of executable or object code is made by offering
+access to copy from a designated place, then offering equivalent
+access to copy the source code from the same place counts as
+distribution of the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+  4. You may not copy, modify, sublicense, or distribute the Program
+except as expressly provided under this License.  Any attempt
+otherwise to copy, modify, sublicense or distribute the Program is
+void, and will automatically terminate your rights under this License.
+However, parties who have received copies, or rights, from you under
+this License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+  5. You are not required to accept this License, since you have not
+signed it.  However, nothing else grants you permission to modify or
+distribute the Program or its derivative works.  These actions are
+prohibited by law if you do not accept this License.  Therefore, by
+modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Program or works based on it.
+
+  6. Each time you redistribute the Program (or any work based on the
+Program), the recipient automatically receives a license from the
+original licensor to copy, distribute or modify the Program subject to
+these terms and conditions.  You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties to
+this License.
+
+  7. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Program at all.  For example, if a patent
+license would not permit royalty-free redistribution of the Program by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system, which is
+implemented by public license practices.  Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+  8. If the distribution and/or use of the Program is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Program under this License
+may add an explicit geographical distribution limitation excluding
+those countries, so that distribution is permitted only in or among
+countries not thus excluded.  In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+  9. The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+Each version is given a distinguishing version number.  If the Program
+specifies a version number of this License which applies to it and "any
+later version", you have the option of following the terms and conditions
+either of that version or of any later version published by the Free
+Software Foundation.  If the Program does not specify a version number of
+this License, you may choose any version ever published by the Free Software
+Foundation.
+
+  10. If you wish to incorporate parts of the Program into other free
+programs whose distribution conditions are different, write to the author
+to ask for permission.  For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this.  Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software and
+of promoting the sharing and reuse of software generally.
+
+			    NO WARRANTY
+
+  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+		     END OF TERMS AND CONDITIONS
+
+	    How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+Also add information on how to contact you by electronic and paper mail.
+
+If the program is interactive, make it output a short notice like this
+when it starts in an interactive mode:
+
+    Gnomovision version 69, Copyright (C) year name of author
+    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, the commands you use may
+be called something other than `show w' and `show c'; they could even be
+mouse-clicks or menu items--whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary.  Here is a sample; alter the names:
+
+  Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+  `Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+  <signature of Ty Coon>, 1 April 1989
+  Ty Coon, President of Vice
+
+This General Public License does not permit incorporating your program into
+proprietary programs.  If your program is a subroutine library, you may
+consider it more useful to permit linking proprietary applications with the
+library.  If this is what you want to do, use the GNU Lesser General
+Public License instead of this License.

bitbake/ChangeLog

@@ -0,0 +1,317 @@
+Changes in Bitbake 1.9.x:
+	- Add PE (Package Epoch) support from Philipp Zabel (pH5)
+	- Treat python functions the same as shell functions for logging
+	- Use TMPDIR/anonfunc as a __anonfunc temp directory (T)
+	- Catch truncated cache file errors
+	- Allow operations other than assignment on flag variables
+	- Add code to handle inter-task dependencies
+	- Fix cache errors when generation dotGraphs
+	- Make sure __inherit_cache is updated before calling include() (from Michael Krelin)
+	- Fix bug when target was in ASSUME_PROVIDED (#2236)
+	- Raise ParseError for filenames with multiple underscores instead of infinitely looping (#2062)
+	- Fix invalid regexp in BBMASK error handling (missing import) (#1124)
+	- Promote certain warnings from debug to note 2 level
+	- Update manual
+	- Correctly redirect stdin when forking
+	- If parsing errors are found, exit, too many users miss the errors
+	- Remove supriours PREFERRED_PROVIDER warnings
+	- svn fetcher: Add _buildsvncommand function
+	- Improve certain error messages
+	- Rewrite svn fetcher to make adding extra operations easier 
+	  as part of future SRCDATE="now" fixes
+	  (requires new FETCHCMD_svn definition in bitbake.conf)
+	- Change SVNDIR layout to be more unique (fixes #2644 and #2624)
+	- Add ConfigParsed Event after configuration parsing is complete
+	- Add SRCREV support for svn fetcher
+	- data.emit_var() - only call getVar if we need the variable
+	- Stop generating the A variable (seems to be legacy code)
+	- Make sure intertask depends get processed correcting in recursive depends
+	- Add pn-PN to overrides when evaluating PREFERRED_VERSION
+	- Improve the progress indicator by skipping tasks that have 
+	  already run before starting the build rather than during it
+	- Add profiling option (-P)
+	- Add BB_SRCREV_POLICY variable (clear or cache) to control SRCREV cache
+	- Add SRCREV_FORMAT support
+	- Fix local fetcher's localpath return values
+	- Apply OVERRIDES before performing immediate expansions
+	- Allow the -b -e option combination to take regular expressions
+	- Fix handling of variables with expansion in the name using _append/_prepend
+	  e.g. RRECOMMENDS_${PN}_append_xyz = "abc"
+	- Add plain message function to bb.msg
+	- Sort the list of providers before processing so dependency problems are 
+	  reproducible rather than effectively random
+	- Fix/improve bitbake -s output
+	- Add locking for fetchers so only one tries to fetch a given file at a given time
+	- Fix int(0)/None confusion in runqueue.py which causes random gaps in dependency chains	  
+	- Expand data in addtasks
+	- Print the list of missing DEPENDS,RDEPENDS for the "No buildable providers available for required...."
+	  error message.
+	- Rework add_task to be more efficient (6% speedup, 7% number of function calls reduction)
+	- Sort digraph output to make builds more reproducible
+	- Split expandKeys into two for loops to benefit from the expand_cache (12% speedup)
+	- runqueue.py: Fix idepends handling to avoid dependency errors
+	- Clear the terminal TOSTOP flag if set (and warn the user)
+	- Fix regression from r653 and make SRCDATE/CVSDATE work for packages again
+	- Fix a bug in bb.decodeurl where http://some.where.com/somefile.tgz decoded to host="" (#1530)
+	- Warn about malformed PREFERRED_PROVIDERS (#1072)
+	- Add support for BB_NICE_LEVEL option (#1627)
+	- Psyco is used only on x86 as there is no support for other architectures.
+	- Sort initial providers list by default preference (#1145, #2024)
+	- Improve provider sorting so prefered versions have preference over latest versions (#768)
+	- Detect builds of tasks with overlapping providers and warn (will become a fatal error) (#1359)
+	- Add MULTI_PROVIDER_WHITELIST variable to allow known safe multiple providers to be listed
+	- Handle paths in svn fetcher module parameter
+	- Support the syntax "export VARIABLE"
+	- Add bzr fetcher
+	- Add support for cleaning directories before a task in the form:
+	  do_taskname[cleandirs] = "dir"
+	- bzr fetcher tweaks from Robert Schuster (#2913)
+	- Add mercurial (hg) fetcher from Robert Schuster (#2913)
+	- Don't add duplicates to BBPATH
+	- Fix preferred_version return values (providers.py)
+	- Fix 'depends' flag splitting
+	- Fix unexport handling (#3135)
+	- Add bb.copyfile function similar to bb.movefile (and improve movefile error reporting)
+	- Allow multiple options for deptask flag
+	- Use git-fetch instead of git-pull removing any need for merges when 
+	  fetching (we don't care about the index). Fixes fetch errors.
+	- Add BB_GENERATE_MIRROR_TARBALLS option, set to 0 to make git fetches 
+	  faster at the expense of not creating mirror tarballs.
+	- SRCREV handling updates, improvements and fixes from Poky
+	- Add bb.utils.lockfile() and bb.utils.unlockfile() from Poky
+	- Add support for task selfstamp and lockfiles flags
+	- Disable task number acceleration since it can allow the tasks to run 
+	  out of sequence
+	- Improve runqueue code comments
+	- Add task scheduler abstraction and some example schedulers
+	- Improve circular dependency chain debugging code and user feedback
+	- Don't give a stacktrace for invalid tasks, have a user friendly message (#3431)
+	- Add support for "-e target" (#3432)
+	- Fix shell showdata command (#3259)
+	- Fix shell data updating problems (#1880)
+	- Properly raise errors for invalid source URI protocols
+	- Change the wget fetcher failure handling to avoid lockfile problems
+	- Add support for branches in git fetcher (Otavio Salvador, Michael Lauer)
+	- Make taskdata and runqueue errors more user friendly
+	- Add norecurse and fullpath options to cvs fetcher
+	- Fix exit code for build failures in --continue mode
+	- Fix git branch tags fetching
+	- Change parseConfigurationFile so it works on real data, not a copy
+	- Handle 'base' inherit and all other INHERITs from parseConfigurationFile 
+	  instead of BBHandler
+	- Fix getVarFlags bug in data_smart
+	- Optmise cache handling by more quickly detecting an invalid cache, only 
+	  saving the cache when its changed, moving the cache validity check into
+	  the parsing loop and factoring some getVar calls outside a for loop
+	- Cooker: Remove a debug message from the parsing loop to lower overhead
+	- Convert build.py exec_task to use getVarFlags
+	- Update shell to use cooker.buildFile
+	- Add StampUpdate event
+	- Convert -b option to use taskdata/runqueue
+	- Remove digraph and switch to new stamp checking code. exec_task no longer
+	  honours dependencies
+	- Make fetcher timestamp updating non-fatal when permissions don't allow 
+	  updates
+	- Add BB_SCHEDULER variable/option ("completion" or "speed") controlling
+	  the way bitbake schedules tasks
+	- Add BB_STAMP_POLICY variable/option ("perfile" or "full") controlling
+	  how extensively stamps are looked at for validity
+	- When handling build target failures make sure idepends are checked and
+	  failed where needed. Fixes --continue mode crashes.
+	- Fix -f (force) in conjunction with -b
+	- Fix problems with recrdeptask handling where some idepends weren't handled
+	  correctly.
+	- Handle exit codes correctly (from pH5)
+	- Work around refs/HEAD issues with git over http (#3410)
+	- Add proxy support to the CVS fetcher (from Cyril Chemparathy)
+	- Improve runfetchcmd so errors are seen and various GIT variables are exported
+	- Add ability to fetchers to check URL validity without downloading
+	- Improve runtime PREFERRED_PROVIDERS warning message
+	- Add BB_STAMP_WHITELIST option which contains a list of stamps to ignore when
+	  checking stamp dependencies and using a BB_STAMP_POLICY of "whitelist"
+	- No longer weight providers on the basis of a package being "already staged". This
+	  leads to builds being non-deterministic.
+	- Flush stdout/stderr before forking to fix duplicate console output
+	- Make sure recrdeps tasks include all inter-task dependencies of a given fn
+	- Add bb.runqueue.check_stamp_fn() for use by packaged-staging
+	- Add PERSISTENT_DIR to store the PersistData in a persistent
+	  directory != the cache dir.
+	- Add md5 and sha256 checksum generation functions to utils.py
+	- Correctly handle '-' characters in class names (#2958)
+	- Make sure expandKeys has been called on the data dictionary before running tasks
+	- Correctly add a task override in the form task-TASKNAME.
+	- Revert the '-' character fix in class names since it breaks things
+	- When a regexp fails to compile for PACKAGES_DYNAMIC, print a more useful error (#4444)
+	- Allow to checkout CVS by Date and Time. Just add HHmm to the SRCDATE.
+	- Move prunedir function to utils.py and add explode_dep_versions function
+	- Raise an exception if SRCREV == 'INVALID'
+	- Fix hg fetcher username/password handling and fix crash
+	- Fix PACKAGES_DYNAMIC handling of packages with '++' in the name
+	- Rename __depends to __base_depends after configuration parsing so we don't
+	  recheck the validity of the config files time after time
+	- Add better environmental variable handling. By default it will now only pass certain 
+	  whitelisted variables into the data store. If BB_PRESERVE_ENV is set bitbake will use
+	  all variable from the environment. If BB_ENV_WHITELIST is set, that whitelist will be
+	  used instead of the internal bitbake one. Alternatively, BB_ENV_EXTRAWHITE can be used
+	  to extend the internal whitelist.
+	- Perforce fetcher fix to use commandline options instead of being overriden by the environment
+	- bb.utils.prunedir can cope with symlinks to directoriees without exceptions
+	- use @rev when doing a svn checkout
+	- Add osc fetcher (from Joshua Lock in Poky)
+	- When SRCREV autorevisioning for a recipe is in use, don't cache the recipe
+	- Add tryaltconfigs option to control whether bitbake trys using alternative providers
+	  to fulfil failed dependencies. It defaults to off, changing the default since this
+	  behaviour confuses many users and isn't often useful.
+	- Improve lock file function error handling
+	- Add username handling to the git fetcher (Robert Bragg)
+	- Add support for HTTP_PROXY and HTTP_PROXY_IGNORE variables to the wget fetcher
+	- Export more variables to the fetcher commands to allow ssh checkouts and checkouts through 
+	  proxies to work better. (from Poky)
+	- Also allow user and pswd options in SRC_URIs globally (from Poky)
+	- Improve proxy handling when using mirrors (from Poky)
+	- Add bb.utils.prune_suffix function
+	- Fix hg checkouts of specific revisions (from Poky)
+	- Fix wget fetching of urls with parameters specified (from Poky)
+	- Add username handling to git fetcher (from Poky)
+	- Set HOME environmental variable when running fetcher commands (from Poky)
+	- Make sure allowed variables inherited from the environment are exported again (from Poky)
+	- When running a stage task in bbshell, run populate_staging, not the stage task (from Poky)
+	- Fix + character escaping from PACKAGES_DYNAMIC (thanks Otavio Salvador)
+	- Addition of BBCLASSEXTEND support for allowing one recipe to provide multiple targets (from Poky)
+
+Changes in Bitbake 1.8.0:
+	- Release 1.7.x as a stable series
+
+Changes in BitBake 1.7.x:
+	- Major updates of the dependency handling and execution
+	  of tasks. Code from bin/bitbake replaced with runqueue.py
+	  and taskdata.py
+	- New task execution code supports multithreading with a simplistic
+	  threading algorithm controlled by BB_NUMBER_THREADS
+	- Change of the SVN Fetcher to keep the checkout around
+	  courtsey of Paul Sokolovsky (#1367)
+	- PATH fix to bbimage (#1108)
+	- Allow debug domains to be specified on the commandline (-l)
+	- Allow 'interactive' tasks
+	- Logging message improvements
+	- Drop now uneeded BUILD_ALL_DEPS variable
+	- Add support for wildcards to -b option
+	- Major overhaul of the fetchers making a large amount of code common
+	  including mirroring code
+	- Fetchers now touch md5 stamps upon access (to show activity)
+	- Fix -f force option when used without -b (long standing bug)
+	- Add expand_cache to data_cache.py, caching expanded data (speedup)
+	- Allow version field in DEPENDS (ignored for now)
+	- Add abort flag support to the shell
+	- Make inherit fail if the class doesn't exist (#1478)
+	- Fix data.emit_env() to expand keynames as well as values
+	- Add ssh fetcher
+	- Add perforce fetcher
+	- Make PREFERRED_PROVIDER_foobar defaults to foobar if available
+	- Share the parser's mtime_cache, reducing the number of stat syscalls
+	- Compile all anonfuncs at once! 
+	  *** Anonfuncs must now use common spacing format ***
+	- Memorise the list of handlers in __BBHANDLERS and tasks in __BBTASKS
+	  This removes 2 million function calls resulting in a 5-10% speedup
+	- Add manpage
+	- Update generateDotGraph to use taskData/runQueue improving accuracy
+	  and also adding a task dependency graph
+	- Fix/standardise on GPLv2 licence
+	- Move most functionality from bin/bitbake to cooker.py and split into
+	  separate funcitons
+	- CVS fetcher: Added support for non-default port
+	- Add BBINCLUDELOGS_LINES, the number of lines to read from any logfile
+	- Drop shebangs from lib/bb scripts
+
+Changes in Bitbake 1.6.0:
+	- Better msg handling
+	- COW dict implementation from Tim Ansell (mithro) leading
+	  to better performance
+	- Speed up of -s
+
+Changes in Bitbake 1.4.4:
+	- SRCDATE now handling courtsey Justin Patrin
+	- #1017 fix to work with rm_work
+
+Changes in BitBake 1.4.2:
+	- Send logs to oe.pastebin.com instead of pastebin.com
+	  fixes #856
+	- Copy the internal bitbake data before building the
+	  dependency graph. This fixes nano not having a
+	  virtual/libc dependency
+	- Allow multiple TARBALL_STASH entries
+	- Cache, check if the directory exists before changing
+	  into it
+	- git speedup cloning by not doing a checkout
+	- allow to have spaces in filenames (.conf, .bb, .bbclass)
+
+Changes in BitBake 1.4.0:
+	- Fix to check both RDEPENDS and RDEPENDS_${PN}
+	- Fix a RDEPENDS parsing bug in utils:explode_deps()
+	- Update git fetcher behaviour to match git changes
+	- ASSUME_PROVIDED allowed to include runtime packages
+	- git fetcher cleanup and efficency improvements
+	- Change the format of the cache
+	- Update usermanual to document the Fetchers
+	- Major changes to caching with a new strategy
+	  giving a major performance increase when reparsing
+	  with few data changes
+
+Changes in BitBake 1.3.3:
+	- Create a new Fetcher module to ease the
+	  development of new Fetchers.
+	  Issue #438 fixed by rpurdie@openedhand.com
+	- Make the Subversion fetcher honor the SRC Date
+	  (CVSDATE).
+	  Issue #555 fixed by chris@openedhand.com
+	- Expand PREFERRED_PROVIDER properly
+	  Issue #436 fixed by rprudie@openedhand.com
+	- Typo fix for Issue #531 by Philipp Zabel for the
+	  BitBake Shell
+	- Introduce a new special variable SRCDATE as
+	  a generic naming to replace CVSDATE.
+	- Introduce a new keyword 'required'. In contrast
+	  to 'include' parsing will fail if a to be included
+	  file can not be found.
+	- Remove hardcoding of the STAMP directory. Patch
+	  courtsey pHilipp Zabel
+	- Track the RDEPENDS of each package (rpurdie@openedhand.com)
+	- Introduce BUILD_ALL_DEPS to build all RDEPENDS. E.g
+	  this is used by the OpenEmbedded Meta Packages.
+	  (rpurdie@openedhand.com).
+
+Changes in BitBake 1.3.2:
+	- reintegration of make.py into BitBake
+	- bbread is gone, use bitbake -e
+	- lots of shell updates and bugfixes
+	- Introduction of the .= and =. operator
+	- Sort variables, keys and groups in bitdoc
+	- Fix regression in the handling of BBCOLLECTIONS
+	- Update the bitbake usermanual
+
+Changes in BitBake 1.3.0:
+	- add bitbake interactive shell (bitbake -i)
+	- refactor bitbake utility in OO style
+	- kill default arguments in methods in the bb.data module
+	- kill default arguments in methods in the bb.fetch module
+	- the http/https/ftp fetcher will fail if the to be 
+	  downloaded file was not found in DL_DIR (this is needed
+	  to avoid unpacking the sourceforge mirror page)
+	- Switch to a cow like data instance for persistent and non
+	  persisting mode (called data_smart.py)
+	- Changed the callback of bb.make.collect_bbfiles to carry
+	  additional parameters
+	- Drastically reduced the amount of needed RAM by not holding
+	  each data instance in memory when using a cache/persistent
+	  storage
+
+Changes in BitBake 1.2.1:
+	The 1.2.1 release is meant as a intermediate release to lay the
+	ground for more radical changes. The most notable changes are:
+
+	- Do not hardcode {}, use bb.data.init() instead if you want to
+	  get a instance of a data class
+	- bb.data.init() is a factory and the old bb.data methods are delegates
+	- Do not use deepcopy use bb.data.createCopy() instead.
+	- Removed default arguments in bb.fetch
+

bitbake/HEADER

@@ -0,0 +1,19 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# <one line to give the program's name and a brief idea of what it does.>
+# Copyright (C) <year>  <name of author>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+

bitbake/LICENSE

@@ -0,0 +1,12 @@
+BitBake is licensed under the GNU General Public License version 2.0. See COPYING for further details.
+
+The following external components are distributed with this software:
+
+* The Toaster Simple UI application is based upon the Django project template, the files of which are covered by the BSD license and are copyright (c) Django Software
+Foundation and individual contributors.
+
+* Twitter Bootstrap (including Glyphicons), redistributed under the Apache License 2.0.
+
+* jQuery is redistributed under the MIT license.
+
+* QUnit is redistributed under the MIT license.

bitbake/bin/bitbake

@@ -0,0 +1,53 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (C) 2003, 2004  Chris Larson
+# Copyright (C) 2003, 2004  Phil Blundell
+# Copyright (C) 2003 - 2005 Michael 'Mickey' Lauer
+# Copyright (C) 2005        Holger Hans Peter Freyther
+# Copyright (C) 2005        ROAD GmbH
+# Copyright (C) 2006        Richard Purdie
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import sys
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)),
+                                'lib'))
+try:
+    import bb
+except RuntimeError as exc:
+    sys.exit(str(exc))
+
+from bb import cookerdata
+from bb.main import bitbake_main, BitBakeConfigParameters, BBMainException
+
+__version__ = "1.30.0"
+
+if __name__ == "__main__":
+    if __version__ != bb.__version__:
+        sys.exit("Bitbake core version and program version mismatch!")
+    try:
+        sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv),
+                              cookerdata.CookerConfiguration()))
+    except BBMainException as err:
+        sys.exit(err)
+    except bb.BBHandledException:
+        sys.exit(1)
+    except Exception:
+        import traceback
+        traceback.print_exc()
+        sys.exit(1)

bitbake/bin/bitbake-diffsigs

@@ -0,0 +1,138 @@
+#!/usr/bin/env python
+
+# bitbake-diffsigs
+# BitBake task signature data comparison utility
+#
+# Copyright (C) 2012-2013 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import sys
+import warnings
+import fnmatch
+import optparse
+import logging
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
+
+import bb.tinfoil
+import bb.siggen
+
+def logger_create(name, output=sys.stderr):
+    logger = logging.getLogger(name)
+    console = logging.StreamHandler(output)
+    format = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
+    if output.isatty():
+        format.enable_color()
+    console.setFormatter(format)
+    logger.addHandler(console)
+    logger.setLevel(logging.INFO)
+    return logger
+
+logger = logger_create('bitbake-diffsigs')
+
+def find_compare_task(bbhandler, pn, taskname):
+    """ Find the most recent signature files for the specified PN/task and compare them """
+
+    def get_hashval(siginfo):
+        if siginfo.endswith('.siginfo'):
+            return siginfo.rpartition(':')[2].partition('_')[0]
+        else:
+            return siginfo.rpartition('.')[2]
+
+    if not hasattr(bb.siggen, 'find_siginfo'):
+        logger.error('Metadata does not support finding signature data files')
+        sys.exit(1)
+
+    if not taskname.startswith('do_'):
+        taskname = 'do_%s' % taskname
+
+    filedates = bb.siggen.find_siginfo(pn, taskname, None, bbhandler.config_data)
+    latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-3:]
+    if not latestfiles:
+        logger.error('No sigdata files found matching %s %s' % (pn, taskname))
+        sys.exit(1)
+    elif len(latestfiles) < 2:
+        logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (pn, taskname))
+        sys.exit(1)
+    else:
+        # It's possible that latestfiles contain 3 elements and the first two have the same hash value.
+        # In this case, we delete the second element.
+        # The above case is actually the most common one. Because we may have sigdata file and siginfo
+        # file having the same hash value. Comparing such two files makes no sense.
+        if len(latestfiles) == 3:
+            hash0 = get_hashval(latestfiles[0])
+            hash1 = get_hashval(latestfiles[1])
+            if hash0 == hash1:
+                latestfiles.pop(1)
+
+        # Define recursion callback
+        def recursecb(key, hash1, hash2):
+            hashes = [hash1, hash2]
+            hashfiles = bb.siggen.find_siginfo(key, None, hashes, bbhandler.config_data)
+
+            recout = []
+            if len(hashfiles) == 2:
+                out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb)
+                recout.extend(list('  ' + l for l in out2))
+            else:
+                recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2))
+
+            return recout
+
+        # Recurse into signature comparison
+        output = bb.siggen.compare_sigfiles(latestfiles[0], latestfiles[1], recursecb)
+        if output:
+            print '\n'.join(output)
+    sys.exit(0)
+
+
+
+parser = optparse.OptionParser(
+    description = "Compares siginfo/sigdata files written out by BitBake",
+    usage = """
+  %prog -t recipename taskname
+  %prog sigdatafile1 sigdatafile2
+  %prog sigdatafile1""")
+
+parser.add_option("-t", "--task",
+        help = "find the signature data files for last two runs of the specified task and compare them",
+        action="store", dest="taskargs", nargs=2, metavar='recipename taskname')
+
+options, args = parser.parse_args(sys.argv)
+
+if options.taskargs:
+    tinfoil = bb.tinfoil.Tinfoil()
+    tinfoil.prepare(config_only = True)
+    find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1])
+else:
+    if len(args) == 1:
+        parser.print_help()
+    else:
+        import cPickle
+        try:
+            if len(args) == 2:
+                output = bb.siggen.dump_sigfile(sys.argv[1])
+            else:
+                output = bb.siggen.compare_sigfiles(sys.argv[1], sys.argv[2])
+        except IOError as e:
+            logger.error(str(e))
+            sys.exit(1)
+        except cPickle.UnpicklingError, EOFError:
+            logger.error('Invalid signature data - ensure you are specifying sigdata/siginfo files')
+            sys.exit(1)
+
+        if output:
+            print '\n'.join(output)

bitbake/bin/bitbake-dumpsig

@@ -0,0 +1,65 @@
+#!/usr/bin/env python
+
+# bitbake-dumpsig
+# BitBake task signature dump utility
+#
+# Copyright (C) 2013 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import sys
+import warnings
+import optparse
+import logging
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
+
+import bb.siggen
+
+def logger_create(name, output=sys.stderr):
+    logger = logging.getLogger(name)
+    console = logging.StreamHandler(output)
+    format = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
+    if output.isatty():
+        format.enable_color()
+    console.setFormatter(format)
+    logger.addHandler(console)
+    logger.setLevel(logging.INFO)
+    return logger
+
+logger = logger_create('bitbake-dumpsig')
+
+parser = optparse.OptionParser(
+    description = "Dumps siginfo/sigdata files written out by BitBake",
+    usage = """
+  %prog sigdatafile""")
+
+options, args = parser.parse_args(sys.argv)
+
+if len(args) == 1:
+    parser.print_help()
+else:
+    import cPickle
+    try:
+        output = bb.siggen.dump_sigfile(args[1])
+    except IOError as e:
+        logger.error(str(e))
+        sys.exit(1)
+    except cPickle.UnpicklingError, EOFError:
+        logger.error('Invalid signature data - ensure you are specifying a sigdata/siginfo file')
+        sys.exit(1)
+
+    if output:
+        print '\n'.join(output)

bitbake/bin/bitbake-prserv

@@ -0,0 +1,55 @@
+#!/usr/bin/env python
+import os
+import sys,logging
+import optparse
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)),'lib'))
+
+import prserv
+import prserv.serv
+
+__version__="1.0.0"
+
+PRHOST_DEFAULT='0.0.0.0'
+PRPORT_DEFAULT=8585
+
+def main():
+    parser = optparse.OptionParser(
+        version="Bitbake PR Service Core version %s, %%prog version %s" % (prserv.__version__, __version__),
+        usage = "%prog < --start | --stop > [options]")
+
+    parser.add_option("-f", "--file", help="database filename(default: prserv.sqlite3)", action="store",
+                      dest="dbfile", type="string", default="prserv.sqlite3")
+    parser.add_option("-l", "--log", help="log filename(default: prserv.log)", action="store",
+                      dest="logfile", type="string", default="prserv.log")
+    parser.add_option("--loglevel", help="logging level, i.e. CRITICAL, ERROR, WARNING, INFO, DEBUG",
+                      action = "store", type="string", dest="loglevel", default = "INFO")
+    parser.add_option("--start", help="start daemon",
+                      action="store_true", dest="start")
+    parser.add_option("--stop", help="stop daemon",
+                      action="store_true", dest="stop")
+    parser.add_option("--host", help="ip address to bind", action="store",
+                      dest="host", type="string", default=PRHOST_DEFAULT)
+    parser.add_option("--port", help="port number(default: 8585)", action="store",
+                      dest="port", type="int", default=PRPORT_DEFAULT)
+
+    options, args = parser.parse_args(sys.argv)
+    prserv.init_logger(os.path.abspath(options.logfile),options.loglevel)
+
+    if options.start:
+        ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile))
+    elif options.stop:
+        ret=prserv.serv.stop_daemon(options.host, options.port)
+    else:
+        ret=parser.print_help()
+    return ret
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+        traceback.print_exc()
+    sys.exit(ret)
+

bitbake/bin/bitbake-selftest

@@ -0,0 +1,55 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2012 Richard Purdie
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import sys, logging
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib'))
+
+import unittest
+try:
+    import bb
+except RuntimeError as exc:
+    sys.exit(str(exc))
+
+def usage():
+    print('usage: [BB_SKIP_NETTESTS=yes] %s [-v] [testname1 [testname2]...]' % os.path.basename(sys.argv[0]))
+
+verbosity = 1
+
+tests = sys.argv[1:]
+if '-v' in sys.argv:
+    tests.remove('-v')
+    verbosity = 2
+
+if tests:
+    if '--help' in sys.argv[1:]:
+        usage()
+        sys.exit(0)
+else:
+    tests = ["bb.tests.codeparser",
+             "bb.tests.cow",
+             "bb.tests.data",
+             "bb.tests.fetch",
+             "bb.tests.parse",
+             "bb.tests.utils"]
+
+for t in tests:
+    t = '.'.join(t.split('.')[:3])
+    __import__(t)
+
+unittest.main(argv=["bitbake-selftest"] + tests, verbosity=verbosity)
+

bitbake/bin/bitbake-worker

@@ -0,0 +1,447 @@
+#!/usr/bin/env python
+
+import os
+import sys
+import warnings
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
+from bb import fetch2
+import logging
+import bb
+import select
+import errno
+import signal
+from multiprocessing import Lock
+
+# Users shouldn't be running this code directly
+if len(sys.argv) != 2 or not sys.argv[1].startswith("decafbad"):
+    print("bitbake-worker is meant for internal execution by bitbake itself, please don't use it standalone.")
+    sys.exit(1)
+
+profiling = False
+if sys.argv[1].startswith("decafbadbad"):
+    profiling = True
+    try:
+        import cProfile as profile
+    except:
+        import profile
+
+# Unbuffer stdout to avoid log truncation in the event
+# of an unorderly exit as well as to provide timely
+# updates to log files for use with tail
+try:
+    if sys.stdout.name == '<stdout>':
+        sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
+except:
+    pass
+
+logger = logging.getLogger("BitBake")
+
+try:
+    import cPickle as pickle
+except ImportError:
+    import pickle
+    bb.msg.note(1, bb.msg.domain.Cache, "Importing cPickle failed. Falling back to a very slow implementation.")
+
+
+worker_pipe = sys.stdout.fileno()
+bb.utils.nonblockingfd(worker_pipe)
+# Need to guard against multiprocessing being used in child processes
+# and multiple processes trying to write to the parent at the same time
+worker_pipe_lock = None
+
+handler = bb.event.LogHandler()
+logger.addHandler(handler)
+
+if 0:
+    # Code to write out a log file of all events passing through the worker
+    logfilename = "/tmp/workerlogfile"
+    format_str = "%(levelname)s: %(message)s"
+    conlogformat = bb.msg.BBLogFormatter(format_str)
+    consolelog = logging.FileHandler(logfilename)
+    bb.msg.addDefaultlogFilter(consolelog)
+    consolelog.setFormatter(conlogformat)
+    logger.addHandler(consolelog)
+
+worker_queue = ""
+
+def worker_fire(event, d):
+    data = "<event>" + pickle.dumps(event) + "</event>"
+    worker_fire_prepickled(data)
+
+def worker_fire_prepickled(event):
+    global worker_queue
+
+    worker_queue = worker_queue + event
+    worker_flush()
+
+def worker_flush():
+    global worker_queue, worker_pipe
+
+    if not worker_queue:
+        return
+
+    try:
+        written = os.write(worker_pipe, worker_queue)
+        worker_queue = worker_queue[written:]
+    except (IOError, OSError) as e:
+        if e.errno != errno.EAGAIN and e.errno != errno.EPIPE:
+            raise
+
+def worker_child_fire(event, d):
+    global worker_pipe
+    global worker_pipe_lock
+
+    data = "<event>" + pickle.dumps(event) + "</event>"
+    try:
+        worker_pipe_lock.acquire()
+        worker_pipe.write(data)
+        worker_pipe_lock.release()
+    except IOError:
+        sigterm_handler(None, None)
+        raise
+
+bb.event.worker_fire = worker_fire
+
+lf = None
+#lf = open("/tmp/workercommandlog", "w+")
+def workerlog_write(msg):
+    if lf:
+        lf.write(msg)
+        lf.flush()
+
+def sigterm_handler(signum, frame):
+    signal.signal(signal.SIGTERM, signal.SIG_DFL)
+    os.killpg(0, signal.SIGTERM)
+    sys.exit()
+
+def fork_off_task(cfg, data, workerdata, fn, task, taskname, appends, taskdepdata, quieterrors=False):
+    # We need to setup the environment BEFORE the fork, since
+    # a fork() or exec*() activates PSEUDO...
+
+    envbackup = {}
+    fakeenv = {}
+    umask = None
+
+    taskdep = workerdata["taskdeps"][fn]
+    if 'umask' in taskdep and taskname in taskdep['umask']:
+        # umask might come in as a number or text string..
+        try:
+             umask = int(taskdep['umask'][taskname],8)
+        except TypeError:
+             umask = taskdep['umask'][taskname]
+
+    # We can't use the fakeroot environment in a dry run as it possibly hasn't been built
+    if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] and not cfg.dry_run:
+        envvars = (workerdata["fakerootenv"][fn] or "").split()
+        for key, value in (var.split('=') for var in envvars):
+            envbackup[key] = os.environ.get(key)
+            os.environ[key] = value
+            fakeenv[key] = value
+
+        fakedirs = (workerdata["fakerootdirs"][fn] or "").split()
+        for p in fakedirs:
+            bb.utils.mkdirhier(p)
+        logger.debug(2, 'Running %s:%s under fakeroot, fakedirs: %s' %
+                        (fn, taskname, ', '.join(fakedirs)))
+    else:
+        envvars = (workerdata["fakerootnoenv"][fn] or "").split()
+        for key, value in (var.split('=') for var in envvars):
+            envbackup[key] = os.environ.get(key)
+            os.environ[key] = value
+            fakeenv[key] = value
+
+    sys.stdout.flush()
+    sys.stderr.flush()
+
+    try:
+        pipein, pipeout = os.pipe()
+        pipein = os.fdopen(pipein, 'rb', 4096)
+        pipeout = os.fdopen(pipeout, 'wb', 0)
+        pid = os.fork()
+    except OSError as e:
+        bb.msg.fatal("RunQueue", "fork failed: %d (%s)" % (e.errno, e.strerror))
+
+    if pid == 0:
+        def child():
+            global worker_pipe
+            global worker_pipe_lock
+            pipein.close()
+
+            signal.signal(signal.SIGTERM, sigterm_handler)
+            # Let SIGHUP exit as SIGTERM
+            signal.signal(signal.SIGHUP, sigterm_handler)
+            bb.utils.signal_on_parent_exit("SIGTERM")
+
+            # Save out the PID so that the event can include it the
+            # events
+            bb.event.worker_pid = os.getpid()
+            bb.event.worker_fire = worker_child_fire
+            worker_pipe = pipeout
+            worker_pipe_lock = Lock()
+
+            # Make the child the process group leader and ensure no
+            # child process will be controlled by the current terminal
+            # This ensures signals sent to the controlling terminal like Ctrl+C
+            # don't stop the child processes.
+            os.setsid()
+            # No stdin
+            newsi = os.open(os.devnull, os.O_RDWR)
+            os.dup2(newsi, sys.stdin.fileno())
+
+            if umask:
+                os.umask(umask)
+
+            data.setVar("BB_WORKERCONTEXT", "1")
+            data.setVar("BB_TASKDEPDATA", taskdepdata)
+            data.setVar("BUILDNAME", workerdata["buildname"])
+            data.setVar("DATE", workerdata["date"])
+            data.setVar("TIME", workerdata["time"])
+            bb.parse.siggen.set_taskdata(workerdata["sigdata"])
+            ret = 0
+            try:
+                the_data = bb.cache.Cache.loadDataFull(fn, appends, data)
+                the_data.setVar('BB_TASKHASH', workerdata["runq_hash"][task])
+
+                bb.utils.set_process_name("%s:%s" % (the_data.getVar("PN", True), taskname.replace("do_", "")))
+
+                # exported_vars() returns a generator which *cannot* be passed to os.environ.update() 
+                # successfully. We also need to unset anything from the environment which shouldn't be there 
+                exports = bb.data.exported_vars(the_data)
+                bb.utils.empty_environment()
+                for e, v in exports:
+                    os.environ[e] = v
+                for e in fakeenv:
+                    os.environ[e] = fakeenv[e]
+                    the_data.setVar(e, fakeenv[e])
+                    the_data.setVarFlag(e, 'export', "1")
+
+                if quieterrors:
+                    the_data.setVarFlag(taskname, "quieterrors", "1")
+
+            except Exception as exc:
+                if not quieterrors:
+                    logger.critical(str(exc))
+                os._exit(1)
+            try:
+                if cfg.dry_run:
+                    return 0
+                return bb.build.exec_task(fn, taskname, the_data, cfg.profile)
+            except:
+                os._exit(1)
+        if not profiling:
+            os._exit(child())
+        else:
+            profname = "profile-%s.log" % (fn.replace("/", "-") + "-" + taskname)
+            prof = profile.Profile()
+            try: 
+                ret = profile.Profile.runcall(prof, child)
+            finally:
+                prof.dump_stats(profname)
+                bb.utils.process_profilelog(profname)
+                os._exit(ret)
+    else:
+        for key, value in envbackup.iteritems():
+            if value is None:
+                del os.environ[key]
+            else:
+                os.environ[key] = value
+
+    return pid, pipein, pipeout
+
+class runQueueWorkerPipe():
+    """
+    Abstraction for a pipe between a worker thread and the worker server
+    """
+    def __init__(self, pipein, pipeout):
+        self.input = pipein
+        if pipeout:
+            pipeout.close()
+        bb.utils.nonblockingfd(self.input)
+        self.queue = ""
+
+    def read(self):
+        start = len(self.queue)
+        try:
+            self.queue = self.queue + self.input.read(102400)
+        except (OSError, IOError) as e:
+            if e.errno != errno.EAGAIN:
+                raise
+
+        end = len(self.queue)
+        index = self.queue.find("</event>")
+        while index != -1:
+            worker_fire_prepickled(self.queue[:index+8])
+            self.queue = self.queue[index+8:]
+            index = self.queue.find("</event>")
+        return (end > start)
+
+    def close(self):
+        while self.read():
+            continue
+        if len(self.queue) > 0:
+            print("Warning, worker child left partial message: %s" % self.queue)
+        self.input.close()
+
+normalexit = False
+
+class BitbakeWorker(object):
+    def __init__(self, din):
+        self.input = din
+        bb.utils.nonblockingfd(self.input)
+        self.queue = ""
+        self.cookercfg = None
+        self.databuilder = None
+        self.data = None
+        self.build_pids = {}
+        self.build_pipes = {}
+    
+        signal.signal(signal.SIGTERM, self.sigterm_exception)
+        # Let SIGHUP exit as SIGTERM
+        signal.signal(signal.SIGHUP, self.sigterm_exception)
+        if "beef" in sys.argv[1]:
+            bb.utils.set_process_name("Worker (Fakeroot)")
+        else:
+            bb.utils.set_process_name("Worker")
+
+    def sigterm_exception(self, signum, stackframe):
+        if signum == signal.SIGTERM:
+            bb.warn("Worker received SIGTERM, shutting down...")
+        elif signum == signal.SIGHUP:
+            bb.warn("Worker received SIGHUP, shutting down...")
+        self.handle_finishnow(None)
+        signal.signal(signal.SIGTERM, signal.SIG_DFL)
+        os.kill(os.getpid(), signal.SIGTERM)
+
+    def serve(self):        
+        while True:
+            (ready, _, _) = select.select([self.input] + [i.input for i in self.build_pipes.values()], [] , [], 1)
+            if self.input in ready:
+                try:
+                    r = self.input.read()
+                    if len(r) == 0:
+                        # EOF on pipe, server must have terminated
+                        self.sigterm_exception(signal.SIGTERM, None)
+                    self.queue = self.queue + r
+                except (OSError, IOError):
+                    pass
+            if len(self.queue):
+                self.handle_item("cookerconfig", self.handle_cookercfg)
+                self.handle_item("workerdata", self.handle_workerdata)
+                self.handle_item("runtask", self.handle_runtask)
+                self.handle_item("finishnow", self.handle_finishnow)
+                self.handle_item("ping", self.handle_ping)
+                self.handle_item("quit", self.handle_quit)
+
+            for pipe in self.build_pipes:
+                self.build_pipes[pipe].read()
+            if len(self.build_pids):
+                self.process_waitpid()
+            worker_flush()
+
+
+    def handle_item(self, item, func):
+        if self.queue.startswith("<" + item + ">"):
+            index = self.queue.find("</" + item + ">")
+            while index != -1:
+                func(self.queue[(len(item) + 2):index])
+                self.queue = self.queue[(index + len(item) + 3):]
+                index = self.queue.find("</" + item + ">")
+
+    def handle_cookercfg(self, data):
+        self.cookercfg = pickle.loads(data)
+        self.databuilder = bb.cookerdata.CookerDataBuilder(self.cookercfg, worker=True)
+        self.databuilder.parseBaseConfiguration()
+        self.data = self.databuilder.data
+
+    def handle_workerdata(self, data):
+        self.workerdata = pickle.loads(data)
+        bb.msg.loggerDefaultDebugLevel = self.workerdata["logdefaultdebug"]
+        bb.msg.loggerDefaultVerbose = self.workerdata["logdefaultverbose"]
+        bb.msg.loggerVerboseLogs = self.workerdata["logdefaultverboselogs"]
+        bb.msg.loggerDefaultDomains = self.workerdata["logdefaultdomain"]
+        self.data.setVar("PRSERV_HOST", self.workerdata["prhost"])
+
+    def handle_ping(self, _):
+        workerlog_write("Handling ping\n")
+
+        logger.warn("Pong from bitbake-worker!")
+
+    def handle_quit(self, data):
+        workerlog_write("Handling quit\n")
+
+        global normalexit
+        normalexit = True
+        sys.exit(0)
+
+    def handle_runtask(self, data):
+        fn, task, taskname, quieterrors, appends, taskdepdata = pickle.loads(data)
+        workerlog_write("Handling runtask %s %s %s\n" % (task, fn, taskname))
+
+        pid, pipein, pipeout = fork_off_task(self.cookercfg, self.data, self.workerdata, fn, task, taskname, appends, taskdepdata, quieterrors)
+
+        self.build_pids[pid] = task
+        self.build_pipes[pid] = runQueueWorkerPipe(pipein, pipeout)
+
+    def process_waitpid(self):
+        """
+        Return none is there are no processes awaiting result collection, otherwise
+        collect the process exit codes and close the information pipe.
+        """
+        try:
+            pid, status = os.waitpid(-1, os.WNOHANG)
+            if pid == 0 or os.WIFSTOPPED(status):
+                return None
+        except OSError:
+            return None
+
+        workerlog_write("Exit code of %s for pid %s\n" % (status, pid))
+
+        if os.WIFEXITED(status):
+            status = os.WEXITSTATUS(status)
+        elif os.WIFSIGNALED(status):
+            # Per shell conventions for $?, when a process exits due to
+            # a signal, we return an exit code of 128 + SIGNUM
+            status = 128 + os.WTERMSIG(status)
+
+        task = self.build_pids[pid]
+        del self.build_pids[pid]
+
+        self.build_pipes[pid].close()
+        del self.build_pipes[pid]
+
+        worker_fire_prepickled("<exitcode>" + pickle.dumps((task, status)) + "</exitcode>")
+
+    def handle_finishnow(self, _):
+        if self.build_pids:
+            logger.info("Sending SIGTERM to remaining %s tasks", len(self.build_pids))
+            for k, v in self.build_pids.iteritems():
+                try:
+                    os.kill(-k, signal.SIGTERM)
+                    os.waitpid(-1, 0)
+                except:
+                    pass
+        for pipe in self.build_pipes:
+            self.build_pipes[pipe].read()
+
+try:
+    worker = BitbakeWorker(sys.stdin)
+    if not profiling:
+        worker.serve()
+    else:
+        profname = "profile-worker.log"
+        prof = profile.Profile()
+        try:
+            profile.Profile.runcall(prof, worker.serve)
+        finally:
+            prof.dump_stats(profname)
+            bb.utils.process_profilelog(profname)
+except BaseException as e:
+    if not normalexit:
+        import traceback
+        sys.stderr.write(traceback.format_exc())
+        sys.stderr.write(str(e))
+while len(worker_queue):
+    worker_flush()
+workerlog_write("exitting")
+sys.exit(0)
+

bitbake/bin/bitdoc

@@ -0,0 +1,531 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (C) 2005 Holger Hans Peter Freyther
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import optparse, os, sys
+
+# bitbake
+sys.path.append(os.path.join(os.path.dirname(os.path.dirname(__file__), 'lib'))
+import bb
+import bb.parse
+from   string import split, join
+
+__version__ = "0.0.2"
+
+class HTMLFormatter:
+    """
+    Simple class to help to generate some sort of HTML files. It is
+    quite inferior solution compared to docbook, gtkdoc, doxygen but it
+    should work for now.
+    We've a global introduction site (index.html) and then one site for
+    the list of keys (alphabetical sorted) and one for the list of groups,
+    one site for each key with links to the relations and groups.
+
+        index.html
+        all_keys.html
+        all_groups.html
+        groupNAME.html
+        keyNAME.html
+    """
+
+    def replace(self, text, *pairs):
+        """
+        From pydoc... almost identical at least
+        """
+        while pairs:
+            (a, b) = pairs[0]
+            text = join(split(text, a), b)
+            pairs = pairs[1:]
+        return text
+    def escape(self, text):
+        """
+        Escape string to be conform HTML
+        """
+        return self.replace(text, 
+                            ('&', '&'), 
+                            ('<', '&lt;' ),
+                            ('>', '&gt;' ) )
+    def createNavigator(self):
+        """
+        Create the navgiator
+        """
+        return """<table class="navigation" width="100%" summary="Navigation header" cellpadding="2" cellspacing="2">
+<tr valign="middle">
+<td><a accesskey="g" href="index.html">Home</a></td>
+<td><a accesskey="n" href="all_groups.html">Groups</a></td>
+<td><a accesskey="u" href="all_keys.html">Keys</a></td>
+</tr></table>
+"""
+
+    def relatedKeys(self, item):
+        """
+        Create HTML to link to foreign keys
+        """
+
+        if len(item.related()) == 0:
+            return ""
+
+        txt = "<p><b>See also:</b><br>"
+        txts = []
+        for it in item.related():
+            txts.append("""<a href="key%(it)s.html">%(it)s</a>""" % vars() )
+
+        return txt + ",".join(txts)
+
+    def groups(self, item):
+        """
+        Create HTML to link to related groups
+        """
+
+        if len(item.groups()) == 0:
+            return ""
+
+
+        txt = "<p><b>See also:</b><br>"
+        txts = []
+        for group in item.groups():
+            txts.append( """<a href="group%s.html">%s</a> """ % (group, group) )
+
+        return txt + ",".join(txts)
+
+
+    def createKeySite(self, item):
+        """
+        Create a site for a key. It contains the header/navigator, a heading,
+        the description, links to related keys and to the groups.
+        """
+
+        return """<!doctype html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
+<html><head><title>Key %s</title></head>
+<link rel="stylesheet" href="style.css" type="text/css">
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+%s
+<h2><span class="refentrytitle">%s</span></h2>
+
+<div class="refsynopsisdiv">
+<h2>Synopsis</h2>
+<p>
+%s
+</p>
+</div>
+
+<div class="refsynopsisdiv">
+<h2>Related Keys</h2>
+<p>
+%s
+</p>
+</div>
+
+<div class="refsynopsisdiv">
+<h2>Groups</h2>
+<p>
+%s
+</p>
+</div>
+
+
+</body>
+"""     % (item.name(), self.createNavigator(), item.name(), 
+           self.escape(item.description()), self.relatedKeys(item), self.groups(item))
+
+    def createGroupsSite(self, doc):
+        """
+        Create the Group Overview site
+        """
+
+        groups = ""
+        sorted_groups = sorted(doc.groups())
+        for group in sorted_groups:
+            groups += """<a href="group%s.html">%s</a><br>""" % (group, group)
+
+        return """<!doctype html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
+<html><head><title>Group overview</title></head>
+<link rel="stylesheet" href="style.css" type="text/css">
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+%s
+<h2>Available Groups</h2>
+%s
+</body>
+""" % (self.createNavigator(), groups)
+
+    def createIndex(self):
+        """
+        Create the index file
+        """
+
+        return """<!doctype html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
+<html><head><title>Bitbake Documentation</title></head>
+<link rel="stylesheet" href="style.css" type="text/css">
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+%s
+<h2>Documentation Entrance</h2>
+<a href="all_groups.html">All available groups</a><br>
+<a href="all_keys.html">All available keys</a><br>
+</body>
+""" % self.createNavigator()
+
+    def createKeysSite(self, doc):
+        """
+        Create Overview of all avilable keys
+        """
+        keys = ""
+        sorted_keys = sorted(doc.doc_keys())
+        for key in sorted_keys:
+            keys += """<a href="key%s.html">%s</a><br>""" % (key, key)
+
+        return """<!doctype html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
+<html><head><title>Key overview</title></head>
+<link rel="stylesheet" href="style.css" type="text/css">
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+%s
+<h2>Available Keys</h2>
+%s
+</body>
+""" % (self.createNavigator(), keys)
+
+    def createGroupSite(self, gr, items, _description = None):
+        """
+        Create a site for a group:
+        Group the name of the group, items contain the name of the keys
+        inside this group
+        """
+        groups = ""
+        description = ""
+
+        # create a section with the group descriptions
+        if _description:
+            description  += "<h2 Description of Grozp %s</h2>" % gr
+            description  += _description
+
+        items.sort(lambda x, y:cmp(x.name(), y.name()))
+        for group in items:
+            groups += """<a href="key%s.html">%s</a><br>""" % (group.name(), group.name())
+
+        return """<!doctype html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
+<html><head><title>Group %s</title></head>
+<link rel="stylesheet" href="style.css" type="text/css">
+<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
+%s
+%s
+<div class="refsynopsisdiv">
+<h2>Keys in Group %s</h2>
+<pre class="synopsis">
+%s
+</pre>
+</div>
+</body>
+""" % (gr, self.createNavigator(), description, gr, groups)
+
+
+
+    def createCSS(self):
+        """
+        Create the CSS file
+        """
+        return """.synopsis, .classsynopsis
+{
+  background: #eeeeee;
+  border: solid 1px #aaaaaa;
+  padding: 0.5em;
+}
+.programlisting
+{
+  background: #eeeeff;
+  border: solid 1px #aaaaff;
+  padding: 0.5em;
+}
+.variablelist
+{
+  padding: 4px;
+  margin-left: 3em;
+}
+.variablelist td:first-child
+{
+  vertical-align: top;
+}
+table.navigation
+{
+  background: #ffeeee;
+  border: solid 1px #ffaaaa;
+  margin-top: 0.5em;
+  margin-bottom: 0.5em;
+}
+.navigation a
+{
+  color: #770000;
+}
+.navigation a:visited
+{
+  color: #550000;
+}
+.navigation .title
+{
+  font-size: 200%;
+}
+div.refnamediv
+{
+  margin-top: 2em;
+}
+div.gallery-float
+{
+  float: left;
+  padding: 10px;
+}
+div.gallery-float img
+{
+  border-style: none;
+}
+div.gallery-spacer
+{
+  clear: both;
+}
+a
+{
+  text-decoration: none;
+}
+a:hover
+{
+  text-decoration: underline;
+  color: #FF0000;
+}
+"""
+
+
+
+class DocumentationItem:
+    """
+    A class to hold information about a configuration
+    item. It contains the key name, description, a list of related names,
+    and the group this item is contained in.
+    """
+
+    def __init__(self):
+        self._groups  = []
+        self._related = []
+        self._name    = ""
+        self._desc    = ""
+
+    def groups(self):
+        return self._groups
+
+    def name(self):
+        return self._name
+
+    def description(self):
+        return self._desc
+
+    def related(self):
+        return self._related
+
+    def setName(self, name):
+        self._name = name
+
+    def setDescription(self, desc):
+        self._desc = desc
+
+    def addGroup(self, group):
+        self._groups.append(group)
+
+    def addRelation(self, relation):
+        self._related.append(relation)
+
+    def sort(self):
+        self._related.sort()
+        self._groups.sort()
+
+
+class Documentation:
+    """
+    Holds the documentation... with mappings from key to items...
+    """
+
+    def __init__(self):
+        self.__keys   = {}
+        self.__groups = {}
+
+    def insert_doc_item(self, item):
+        """
+        Insert the Doc Item into the internal list
+        of representation
+        """
+        item.sort()
+        self.__keys[item.name()] = item
+
+        for group in item.groups():
+            if not group in self.__groups:
+                self.__groups[group] = []
+            self.__groups[group].append(item)
+            self.__groups[group].sort()
+
+
+    def doc_item(self, key):
+        """
+        Return the DocumentationInstance describing the key
+        """
+        try:
+            return self.__keys[key]
+        except KeyError:
+            return None
+
+    def doc_keys(self):
+        """
+        Return the documented KEYS (names)
+        """
+        return self.__keys.keys()
+
+    def groups(self):
+        """
+        Return the names of available groups
+        """
+        return self.__groups.keys()
+
+    def group_content(self, group_name):
+        """
+        Return a list of keys/names that are in a specefic
+        group or the empty list
+        """
+        try:
+            return self.__groups[group_name]
+        except KeyError:
+            return []
+
+
+def parse_cmdline(args):
+    """
+    Parse the CMD line and return the result as a n-tuple
+    """
+
+    parser = optparse.OptionParser( version = "Bitbake Documentation Tool Core version %s, %%prog version %s" % (bb.__version__, __version__))
+    usage  = """%prog [options]
+
+Create a set of html pages (documentation) for a bitbake.conf....
+"""
+
+    # Add the needed options
+    parser.add_option( "-c", "--config", help = "Use the specified configuration file as source",
+                       action = "store", dest = "config", default = os.path.join("conf", "documentation.conf") )
+
+    parser.add_option( "-o", "--output", help = "Output directory for html files",
+                       action = "store", dest = "output", default = "html/" )
+
+    parser.add_option( "-D",  "--debug", help = "Increase the debug level",
+                       action = "count", dest = "debug", default = 0 )
+
+    parser.add_option( "-v", "--verbose", help = "output more chit-char to the terminal",
+                       action = "store_true", dest = "verbose", default = False )
+
+    options, args = parser.parse_args( sys.argv )
+ 
+    bb.msg.init_msgconfig(options.verbose, options.debug)
+
+    return options.config, options.output
+
+def main():
+    """
+    The main Method
+    """
+
+    (config_file, output_dir) = parse_cmdline( sys.argv )
+
+    # right to let us load the file now
+    try:
+        documentation = bb.parse.handle( config_file, bb.data.init() )
+    except IOError:
+        bb.fatal( "Unable to open %s" % config_file )
+    except bb.parse.ParseError:
+        bb.fatal( "Unable to parse %s" % config_file )
+
+    if isinstance(documentation, dict):
+        documentation = documentation[""]
+
+    # Assuming we've the file loaded now, we will initialize the 'tree'
+    doc = Documentation()
+
+    # defined states
+    state_begin = 0
+    state_see   = 1
+    state_group = 2
+
+    for key in bb.data.keys(documentation):
+        data   = documentation.getVarFlag(key, "doc", False)
+        if not data:
+            continue
+
+        # The Documentation now starts
+        doc_ins = DocumentationItem()
+        doc_ins.setName(key)
+
+
+        tokens = data.split(' ')
+        state = state_begin
+        string= ""
+        for token in tokens:
+            token = token.strip(',')
+
+            if not state == state_see and token == "@see":
+                state = state_see
+                continue
+            elif not state == state_group and token  == "@group":
+                state = state_group
+                continue
+
+            if state == state_begin:
+                string += " %s" % token
+            elif state == state_see:
+                doc_ins.addRelation(token)
+            elif state == state_group:
+                doc_ins.addGroup(token)
+
+        # set the description
+        doc_ins.setDescription(string)
+        doc.insert_doc_item(doc_ins)
+
+    # let us create the HTML now
+    bb.utils.mkdirhier(output_dir)
+    os.chdir(output_dir)
+
+    # Let us create the sites now. We do it in the following order
+    # Start with the index.html. It will point to sites explaining all
+    # keys and groups
+    html_slave = HTMLFormatter()
+
+    f = file('style.css', 'w')
+    print >> f, html_slave.createCSS()
+
+    f = file('index.html', 'w')
+    print >> f, html_slave.createIndex()
+
+    f = file('all_groups.html', 'w')
+    print >> f, html_slave.createGroupsSite(doc)
+
+    f = file('all_keys.html', 'w')
+    print >> f, html_slave.createKeysSite(doc)
+
+    # now for each group create the site
+    for group in doc.groups():
+        f = file('group%s.html' % group, 'w')
+        print >> f, html_slave.createGroupSite(group, doc.group_content(group))
+
+    # now for the keys
+    for key in doc.doc_keys():
+        f = file('key%s.html' % doc.doc_item(key).name(), 'w')
+        print >> f, html_slave.createKeySite(doc.doc_item(key))
+
+
+if __name__ == "__main__":
+    main()

bitbake/bin/image-writer

@@ -0,0 +1,122 @@
+#!/usr/bin/env python
+
+# Copyright (c) 2012 Wind River Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import os
+import sys
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname( \
+                                    os.path.abspath(__file__))), 'lib'))
+try:
+    import bb
+except RuntimeError as exc:
+    sys.exit(str(exc))
+
+import gtk
+import optparse
+import pygtk
+
+from bb.ui.crumbs.hobwidget import HobAltButton, HobButton
+from bb.ui.crumbs.hig.crumbsmessagedialog import CrumbsMessageDialog
+from bb.ui.crumbs.hig.deployimagedialog import DeployImageDialog
+from bb.ui.crumbs.hig.imageselectiondialog import ImageSelectionDialog
+
+# I put all the fs bitbake supported here. Need more test.
+DEPLOYABLE_IMAGE_TYPES = ["jffs2", "cramfs", "ext2", "ext3", "ext4", "btrfs", "squashfs", "ubi", "vmdk"]
+Title = "USB Image Writer"
+
+class DeployWindow(gtk.Window):
+    def __init__(self, image_path=''):
+        super(DeployWindow, self).__init__()
+
+        if len(image_path) > 0:
+            valid = True
+            if not os.path.exists(image_path):
+                valid = False
+                lbl = "<b>Invalid image file path: %s.</b>\nPress <b>Select Image</b> to select an image." % image_path
+            else:
+                image_path = os.path.abspath(image_path)
+                extend_name = os.path.splitext(image_path)[1][1:]
+                if extend_name not in DEPLOYABLE_IMAGE_TYPES:
+                    valid = False
+                    lbl = "<b>Undeployable imge type: %s</b>\nPress <b>Select Image</b> to select an image." % extend_name
+
+            if not valid:
+                image_path = ''
+                crumbs_dialog = CrumbsMessageDialog(self, lbl, gtk.STOCK_DIALOG_INFO)
+                button = crumbs_dialog.add_button("Close", gtk.RESPONSE_OK)
+                HobButton.style_button(button)
+                crumbs_dialog.run()
+                crumbs_dialog.destroy()
+
+        self.deploy_dialog = DeployImageDialog(Title, image_path, self,
+                                        gtk.DIALOG_MODAL | gtk.DIALOG_DESTROY_WITH_PARENT
+                                        | gtk.DIALOG_NO_SEPARATOR, None, standalone=True)
+        close_button = self.deploy_dialog.add_button("Close", gtk.RESPONSE_NO)
+        HobAltButton.style_button(close_button)
+        close_button.connect('clicked', gtk.main_quit)
+
+        write_button = self.deploy_dialog.add_button("Write USB image", gtk.RESPONSE_YES)
+        HobAltButton.style_button(write_button)
+
+        self.deploy_dialog.connect('select_image_clicked', self.select_image_clicked_cb)
+        self.deploy_dialog.connect('destroy', gtk.main_quit)
+        response = self.deploy_dialog.show()
+
+    def select_image_clicked_cb(self, dialog):
+        cwd = os.getcwd()
+        dialog = ImageSelectionDialog(cwd, DEPLOYABLE_IMAGE_TYPES, Title, self, gtk.FILE_CHOOSER_ACTION_SAVE )
+        button = dialog.add_button("Cancel", gtk.RESPONSE_NO)
+        HobAltButton.style_button(button)
+        button = dialog.add_button("Open", gtk.RESPONSE_YES)
+        HobAltButton.style_button(button)
+        response = dialog.run()
+
+        if response == gtk.RESPONSE_YES:
+            if not dialog.image_names:
+                lbl = "<b>No selections made</b>\nClicked the radio button to select a image."
+                crumbs_dialog = CrumbsMessageDialog(self, lbl, gtk.STOCK_DIALOG_INFO)
+                button = crumbs_dialog.add_button("Close", gtk.RESPONSE_OK)
+                HobButton.style_button(button)
+                crumbs_dialog.run()
+                crumbs_dialog.destroy()
+                dialog.destroy()
+                return
+
+            # get the full path of image
+            image_path = os.path.join(dialog.image_folder, dialog.image_names[0])
+            self.deploy_dialog.set_image_text_buffer(image_path)
+            self.deploy_dialog.set_image_path(image_path)
+
+        dialog.destroy()
+
+def main():
+    parser = optparse.OptionParser(
+                usage = """%prog [-h] [image_file]
+
+%prog writes bootable images to USB devices. You can
+provide the image file on the command line or select it using the GUI.""")
+
+    options, args = parser.parse_args(sys.argv)
+    image_file = args[1] if len(args) > 1 else ''
+    dw = DeployWindow(image_file)
+
+if __name__ == '__main__':
+    try:
+        main()
+        gtk.main()
+    except Exception:
+        import traceback
+        traceback.print_exc()

bitbake/bin/toaster

@@ -0,0 +1,276 @@
+#!/bin/echo ERROR: This script needs to be sourced. Please run as .
+
+# toaster - shell script to start Toaster
+
+# Copyright (C) 2013-2015 Intel Corp.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see http://www.gnu.org/licenses/.
+
+# Usage: source toaster [start|stop]
+#                       [webport=<port>] [noui] [noweb]
+
+# Helper function to kill a background toaster development server
+
+webserverKillAll()
+{
+    local pidfile
+    for pidfile in ${BUILDDIR}/.toastermain.pid ${BUILDDIR}/.runbuilds.pid; do
+        if [ -f ${pidfile} ]; then
+            pid=`cat ${pidfile}`
+            while kill -0 $pid 2>/dev/null; do
+                kill -SIGTERM -$pid 2>/dev/null
+                sleep 1
+                # Kill processes if they are still running - may happen
+                # in interactive shells
+                ps fux | grep "python.*manage.py runserver" | awk '{print $2}' | xargs kill
+            done
+            rm  ${pidfile}
+        fi
+    done
+}
+
+webserverStartAll()
+{
+    # do not start if toastermain points to a valid process
+    if ! cat "${BUILDDIR}/.toastermain.pid" 2>/dev/null | xargs -I{} kill -0 {} ; then
+        retval=1
+        rm "${BUILDDIR}/.toastermain.pid"
+    fi
+
+    retval=0
+    # you can always add a superuser later via
+    # ../bitbake/lib/toaster/manage.py createsuperuser --username=<ME>
+    $MANAGE migrate --noinput || retval=1
+
+    if [ $retval -eq 1 ]; then
+        echo "Failed migrations, aborting system start" 1>&2
+        return $retval
+    fi
+
+    $MANAGE checksettings --traceback || retval=1
+
+    if [ $retval -eq 1 ]; then
+        printf "\nError while checking settings; aborting\n"
+        return $retval
+    fi
+
+    echo "Starting webserver..."
+
+    $MANAGE runserver "0.0.0.0:$WEB_PORT" \
+           </dev/null >>${BUILDDIR}/toaster_web.log 2>&1 \
+           & echo $! >${BUILDDIR}/.toastermain.pid
+
+    sleep 1
+
+    if ! cat "${BUILDDIR}/.toastermain.pid" | xargs -I{} kill -0 {} ; then
+        retval=1
+        rm "${BUILDDIR}/.toastermain.pid"
+    else
+        echo "Webserver address:  http://0.0.0.0:$WEB_PORT/"
+    fi
+
+    return $retval
+}
+
+INSTOPSYSTEM=0
+
+# define the stop command
+stop_system()
+{
+    # prevent reentry
+    if [ $INSTOPSYSTEM -eq 1 ]; then return; fi
+    INSTOPSYSTEM=1
+    if [ -f ${BUILDDIR}/.toasterui.pid ]; then
+        kill `cat ${BUILDDIR}/.toasterui.pid` 2>/dev/null
+        rm ${BUILDDIR}/.toasterui.pid
+    fi
+    webserverKillAll
+    # unset exported variables
+    unset DATABASE_URL
+    unset TOASTER_CONF
+    unset TOASTER_DIR
+    unset BITBAKE_UI
+    unset BBBASEDIR
+    trap - SIGHUP
+    #trap - SIGCHLD
+    INSTOPSYSTEM=0
+}
+
+verify_prereq() {
+    # Verify Django version
+    reqfile=$(python -c "import os; print os.path.realpath('$BBBASEDIR/toaster-requirements.txt')")
+    exp='s/Django\([><=]\+\)\([^,]\+\),\([><=]\+\)\(.\+\)/'
+    exp=$exp'import sys,django;version=django.get_version().split(".");'
+    exp=$exp'sys.exit(not (version \1 "\2".split(".") and version \3 "\4".split(".")))/p'
+    if ! sed -n "$exp" $reqfile | python - ; then
+        req=`grep ^Django $reqfile`
+        echo "This program needs $req"
+        echo "Please install with pip install -r $reqfile"
+        return 2
+    fi
+
+    return 0
+}
+
+# read command line parameters
+if [ -n "$BASH_SOURCE" ] ; then
+    TOASTER=${BASH_SOURCE}
+elif [ -n "$ZSH_NAME" ] ; then
+    TOASTER=${(%):-%x}
+else
+    TOASTER=$0
+fi
+
+export BBBASEDIR=`dirname $TOASTER`/..
+MANAGE=$BBBASEDIR/lib/toaster/manage.py
+OEROOT=`dirname $TOASTER`/../..
+
+# this is the configuraton file we are using for toaster
+# we are using the same logic that oe-setup-builddir uses
+# (based on TEMPLATECONF and .templateconf) to determine
+# which toasterconf.json to use.
+# note: There are a number of relative path assumptions
+# in the local layers that currently make using an arbitrary
+# toasterconf.json difficult.
+
+. $OEROOT/.templateconf
+if [ -n "$TEMPLATECONF" ]; then
+    if [ ! -d "$TEMPLATECONF" ]; then
+        # Allow TEMPLATECONF=meta-xyz/conf as a shortcut
+        if [ -d "$OEROOT/$TEMPLATECONF" ]; then
+            TEMPLATECONF="$OEROOT/$TEMPLATECONF"
+        fi
+        if [ ! -d "$TEMPLATECONF" ]; then
+            echo >&2 "Error: '$TEMPLATECONF' must be a directory containing toasterconf.json"
+            return 1
+        fi
+    fi
+fi
+
+if [ "$TOASTER_CONF" = "" ]; then
+    TOASTER_CONF="$TEMPLATECONF/toasterconf.json"
+    export TOASTER_CONF=$(python -c "import os; print os.path.realpath('$TOASTER_CONF')")
+fi
+
+if [ ! -f $TOASTER_CONF ]; then
+    echo "$TOASTER_CONF configuration file not found. Set TOASTER_CONF to specify file or fix .templateconf"
+    return 1
+fi
+
+# this defines the dir toaster will use for
+# 1) clones of layers (in _toaster_clones )
+# 2) the build dir (in build)
+# 3) the sqlite db if that is being used.
+# 4) pid's we need to clean up on exit/shutdown
+# note: for future. in order to make this an arbitrary directory, we need to
+# make sure that the toaster.sqlite file doesn't default to `pwd` like it currently does.
+export TOASTER_DIR=`pwd`
+
+WEBSERVER=1
+WEB_PORT="8000"
+unset CMD
+for param in $*; do
+    case $param in
+    noweb )
+            WEBSERVER=0
+    ;;
+    start )
+            CMD=$param
+    ;;
+    stop )
+            CMD=$param
+    ;;
+    webport=*)
+            WEB_PORT="${param#*=}"
+    esac
+done
+
+if [ `basename \"$0\"` = `basename \"${TOASTER}\"` ]; then
+    echo "Error: This script needs to be sourced. Please run as . $TOASTER"
+    return 1
+fi
+
+verify_prereq || return 1
+
+# We make sure we're running in the current shell and in a good environment
+if [ -z "$BUILDDIR" ] ||  ! which bitbake >/dev/null 2>&1 ; then
+    echo "Error: Build environment is not setup or bitbake is not in path." 1>&2
+    return 2
+fi
+
+# this defines the dir toaster will use for
+# 1) clones of layers (in _toaster_clones )
+# 2) the build dir (in build)
+# 3) the sqlite db if that is being used.
+# 4) pid's we need to clean up on exit/shutdown
+# note: for future. in order to make this an arbitrary directory, we need to
+# make sure that the toaster.sqlite file doesn't default to `pwd`
+# like it currently does.
+export TOASTER_DIR=`dirname $BUILDDIR`
+
+# Determine the action. If specified by arguments, fine, if not, toggle it
+if [ "$CMD" = "start" ] ; then
+    if [ -n "$BBSERVER" ]; then
+	echo " Toaster is already running. Exiting..."
+	return 1
+fi
+elif [ "$CMD" = "" ]; then
+    if [ -z "$BBSERVER" ]; then
+        CMD="start"
+    else
+        CMD="stop"
+    fi
+fi
+
+echo "The system will $CMD."
+
+# Execute the commands
+
+case $CMD in
+    start )
+        # check if addr:port is not in use
+        if [ "$CMD" == 'start' ]; then
+             $MANAGE checksocket "0.0.0.0:$WEB_PORT" || return 1
+        fi
+
+        # kill Toaster web server if it's alive
+        if [ -e $BUILDDIR/.toastermain.pid ] && kill -0 `cat $BUILDDIR/.toastermain.pid`; then
+            echo "Warning: bitbake appears to be dead, but the Toaster web server is running." 1>&2
+            echo " Something fishy is going on." 1>&2
+            echo "Cleaning up the web server to start from a clean slate."
+            webserverKillAll
+        fi
+
+        # Create configuration file
+        conf=${BUILDDIR}/conf/local.conf
+        line='INHERIT+="toaster buildhistory"'
+        grep -q "$line" $conf || echo $line >> $conf
+
+        if [ $WEBSERVER -gt 0 ] && ! webserverStartAll; then
+            echo "Failed ${CMD}."
+            return 4
+        fi
+        export BITBAKE_UI='toasterui'
+        export DATABASE_URL=`$MANAGE get-dburl`
+        $MANAGE runbuilds & echo $! >${BUILDDIR}/.runbuilds.pid
+        # set fail safe stop system on terminal exit
+        trap stop_system SIGHUP
+        echo "Successful ${CMD}."
+        return 0
+    ;;
+    stop )
+        stop_system
+        echo "Successful ${CMD}."
+    ;;
+esac

bitbake/bin/toaster-eventreplay

@@ -0,0 +1,174 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (C) 2014        Alex Damian
+#
+# This file re-uses code spread throughout other Bitbake source files.
+# As such, all other copyrights belong to their own right holders.
+#
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+
+# This command takes a filename as a single parameter. The filename is read
+# as a build eventlog, and the ToasterUI is used to process events in the file
+# and log data in the database
+
+from __future__ import print_function
+import os
+import sys, logging
+
+# mangle syspath to allow easy import of modules
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+                                'lib'))
+
+
+import bb.cooker
+from bb.ui import toasterui
+import sys
+import logging
+
+import json, pickle
+
+
+class FileReadEventsServerConnection():
+    """  Emulates a connection to a bitbake server that feeds
+        events coming actually read from a saved log file.
+    """
+
+    class MockConnection():
+        """ fill-in for the proxy to the server. we just return generic data
+        """
+        def __init__(self, sc):
+            self._sc = sc
+
+        def runCommand(self, commandArray):
+            """ emulates running a command on the server; only read-only commands are accepted """
+            command_name = commandArray[0]
+
+            if command_name == "getVariable":
+                if commandArray[1] in self._sc._variables:
+                    return (self._sc._variables[commandArray[1]]['v'], None)
+                return (None, "Missing variable")
+
+            elif command_name == "getAllKeysWithFlags":
+                dump = {}
+                flaglist = commandArray[1]
+                for k in self._sc._variables.keys():
+                    try:
+                        if not k.startswith("__"):
+                            v = self._sc._variables[k]['v']
+                            dump[k] = {
+                                'v' : v ,
+                                'history' : self._sc._variables[k]['history'],
+                            }
+                            for d in flaglist:
+                                dump[k][d] = self._sc._variables[k][d]
+                    except Exception as e:
+                        print(e)
+                return (dump, None)
+            else:
+                raise Exception("Command %s not implemented" % commandArray[0])
+
+        def terminateServer(self):
+            """ do not do anything """
+            pass
+
+
+
+    class EventReader():
+        def __init__(self, sc):
+            self._sc = sc
+            self.firstraise = 0
+
+        def _create_event(self, line):
+            def _import_class(name):
+                assert len(name) > 0
+                assert "." in name, name
+
+                components = name.strip().split(".")
+                modulename = ".".join(components[:-1])
+                moduleklass = components[-1]
+
+                module = __import__(modulename, fromlist=[str(moduleklass)])
+                return getattr(module, moduleklass)
+
+            # we build a toaster event out of current event log line
+            try:
+                event_data = json.loads(line.strip())
+                event_class = _import_class(event_data['class'])
+                event_object = pickle.loads(json.loads(event_data['vars']))
+            except ValueError as e:
+                print("Failed loading ", line)
+                raise e
+
+            if not isinstance(event_object, event_class):
+                raise Exception("Error loading objects %s class %s ", event_object, event_class)
+
+            return event_object
+
+        def waitEvent(self, timeout):
+
+            nextline = self._sc._eventfile.readline()
+            if len(nextline) == 0:
+                # the build data ended, while toasterui still waits for events.
+                # this happens when the server was abruptly stopped, so we simulate this
+                self.firstraise += 1
+                if self.firstraise == 1:
+                    raise KeyboardInterrupt()
+                else:
+                    return None
+            else:
+                self._sc.lineno += 1
+            return self._create_event(nextline)
+
+
+    def _readVariables(self, variableline):
+        self._variables = json.loads(variableline.strip())['allvariables']
+
+
+    def __init__(self, file_name):
+        self.connection = FileReadEventsServerConnection.MockConnection(self)
+        self._eventfile = open(file_name, "r")
+
+        # we expect to have the variable dump at the start of the file
+        self.lineno = 1
+        self._readVariables(self._eventfile.readline())
+
+        self.events = FileReadEventsServerConnection.EventReader(self)
+
+
+
+
+
+class MockConfigParameters():
+    """ stand-in for cookerdata.ConfigParameters; as we don't really config a cooker, this
+        serves just to supply needed interfaces for the toaster ui to work """
+    def __init__(self):
+        self.observe_only = True            # we can only read files
+
+
+# run toaster ui on our mock bitbake class
+if __name__ == "__main__":
+    if len(sys.argv) < 2:
+        print("Usage: %s event.log " % sys.argv[0])
+        sys.exit(1)
+
+    file_name = sys.argv[-1]
+    mock_connection = FileReadEventsServerConnection(file_name)
+    configParams = MockConfigParameters()
+
+    # run the main program and set exit code to the returned value
+    sys.exit(toasterui.main(mock_connection.connection, mock_connection.events, configParams))

bitbake/contrib/README

@@ -0,0 +1 @@
+This directory is for additional contributed files which may be useful.

bitbake/contrib/bbdev.sh

@@ -0,0 +1,31 @@
+# This is a shell function to be sourced into your shell or placed in your .profile,
+# which makes setting things up for BitBake a bit easier.
+#
+# The author disclaims copyright to the contents of this file and places it in the
+# public domain.
+
+bbdev () {
+	local BBDIR PKGDIR BUILDDIR
+	if test x"$1" = "x--help"; then echo >&2 "syntax: bbdev [bbdir [pkgdir [builddir]]]"; return 1; fi
+	if test x"$1" = x; then BBDIR=`pwd`; else BBDIR=$1; fi
+	if test x"$2" = x; then PKGDIR=`pwd`; else PKGDIR=$2; fi
+	if test x"$3" = x; then BUILDDIR=`pwd`; else BUILDDIR=$3; fi
+
+	BBDIR=`readlink -f $BBDIR`
+	PKGDIR=`readlink -f $PKGDIR`
+	BUILDDIR=`readlink -f $BUILDDIR`
+	if ! (test -d $BBDIR && test -d $PKGDIR && test -d $BUILDDIR); then
+		echo >&2 "syntax: bbdev [bbdir [pkgdir [builddir]]]"
+		return 1
+	fi
+	
+	PATH=$BBDIR/bin:$PATH
+	BBPATH=$BBDIR
+	if test x"$BBDIR" != x"$PKGDIR"; then
+		BBPATH=$PKGDIR:$BBPATH
+	fi
+	if test x"$PKGDIR" != x"$BUILDDIR"; then
+		BBPATH=$BUILDDIR:$BBPATH
+	fi
+	export BBPATH
+}

bitbake/contrib/dump_cache.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (C) 2012 Wind River Systems, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+#
+# This is used for dumping the bb_cache.dat, the output format is:
+# recipe_path PN PV PACKAGES
+#
+import os
+import sys
+import warnings
+
+# For importing bb.cache
+sys.path.insert(0, os.path.join(os.path.abspath(os.path.dirname(sys.argv[0])), '../lib'))
+from bb.cache import CoreRecipeInfo
+
+import cPickle as pickle
+
+def main(argv=None):
+    """
+    Get the mapping for the target recipe.
+    """
+    if len(argv) != 1:
+        print >>sys.stderr, "Error, need one argument!"
+        return 2
+
+    cachefile = argv[0]
+
+    with open(cachefile, "rb") as cachefile:
+        pickled = pickle.Unpickler(cachefile)
+        while cachefile:
+            try:
+                key = pickled.load()
+                val = pickled.load()
+            except Exception:
+                break
+            if isinstance(val, CoreRecipeInfo) and (not val.skipped):
+                pn = val.pn
+                # Filter out the native recipes.
+                if key.startswith('virtual:native:') or pn.endswith("-native"):
+                    continue
+
+                # 1.0 is the default version for a no PV recipe.
+                if val.__dict__.has_key("pv"):
+                    pv = val.pv
+                else:
+                    pv = "1.0"
+
+                print("%s %s %s %s" % (key, pn, pv, ' '.join(val.packages)))
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))
+

bitbake/contrib/vim/ftdetect/bitbake.vim

@@ -0,0 +1,24 @@
+" Vim filetype detection file
+" Language:     BitBake
+" Author:       Ricardo Salveti <rsalveti@rsalveti.net>
+" Copyright:    Copyright (C) 2008  Ricardo Salveti <rsalveti@rsalveti.net>
+" Licence:      You may redistribute this under the same terms as Vim itself
+"
+" This sets up the syntax highlighting for BitBake files, like .bb, .bbclass and .inc
+
+if &compatible || version < 600
+    finish
+endif
+
+" .bb, .bbappend and .bbclass
+au BufNewFile,BufRead *.{bb,bbappend,bbclass}	set filetype=bitbake
+
+" .inc
+au BufNewFile,BufRead *.inc		set filetype=bitbake
+
+" .conf
+au BufNewFile,BufRead *.conf
+    \ if (match(expand("%:p:h"), "conf") > 0) |
+    \     set filetype=bitbake |
+    \ endif
+

bitbake/contrib/vim/ftplugin/bitbake.vim

@@ -0,0 +1,2 @@
+set sts=4 sw=4 et
+set cms=#%s

bitbake/contrib/vim/plugin/newbb.vim

@@ -0,0 +1,84 @@
+" Vim plugin file
+" Purpose:	Create a template for new bb files
+" Author:	Ricardo Salveti <rsalveti@gmail.com>
+" Copyright:	Copyright (C) 2008 Ricardo Salveti <rsalveti@gmail.com>
+"
+" This file is licensed under the MIT license, see COPYING.MIT in
+" this source distribution for the terms.
+"
+" Based on the gentoo-syntax package
+"
+" Will try to use git to find the user name and email
+
+if &compatible || v:version < 600
+    finish
+endif
+
+fun! <SID>GetUserName()
+    let l:user_name = system("git config --get user.name")
+    if v:shell_error
+        return "Unknown User"
+    else
+        return substitute(l:user_name, "\n", "", "")
+endfun
+
+fun! <SID>GetUserEmail()
+    let l:user_email = system("git config --get user.email")
+    if v:shell_error
+        return "unknow@user.org"
+    else
+        return substitute(l:user_email, "\n", "", "")
+endfun
+
+fun! BBHeader()
+    let l:current_year = strftime("%Y")
+    let l:user_name = <SID>GetUserName()
+    let l:user_email = <SID>GetUserEmail()
+    0 put ='# Copyright (C) ' . l:current_year .
+                \ ' ' . l:user_name . ' <' . l:user_email . '>'
+    put ='# Released under the MIT license (see COPYING.MIT for the terms)'
+    $
+endfun
+
+fun! NewBBTemplate()
+    let l:paste = &paste
+    set nopaste
+    
+    " Get the header
+    call BBHeader()
+
+    " New the bb template
+    put ='DESCRIPTION = \"\"'
+    put ='HOMEPAGE = \"\"'
+    put ='LICENSE = \"\"' 
+    put ='SECTION = \"\"'
+    put ='DEPENDS = \"\"'
+    put =''
+    put ='SRC_URI = \"\"'
+
+    " Go to the first place to edit
+    0
+    /^DESCRIPTION =/
+    exec "normal 2f\""
+
+    if paste == 1
+        set paste
+    endif
+endfun
+
+if !exists("g:bb_create_on_empty")
+    let g:bb_create_on_empty = 1
+endif
+
+" disable in case of vimdiff
+if v:progname =~ "vimdiff"
+    let g:bb_create_on_empty = 0
+endif
+
+augroup NewBB
+    au BufNewFile *.bb
+                \ if g:bb_create_on_empty |
+                \    call NewBBTemplate() |
+                \ endif
+augroup END
+

bitbake/contrib/vim/syntax/bitbake.vim

@@ -0,0 +1,126 @@
+" Vim syntax file
+" Language:     BitBake bb/bbclasses/inc
+" Author:       Chris Larson <kergoth@handhelds.org>
+"               Ricardo Salveti <rsalveti@rsalveti.net>
+" Copyright:    Copyright (C) 2004  Chris Larson <kergoth@handhelds.org>
+"               Copyright (C) 2008  Ricardo Salveti <rsalveti@rsalveti.net>
+"
+" This file is licensed under the MIT license, see COPYING.MIT in
+" this source distribution for the terms.
+"
+" Syntax highlighting for bb, bbclasses and inc files.
+"
+" It's an entirely new type, just has specific syntax in shell and python code
+
+if &compatible || v:version < 600
+    finish
+endif
+if exists("b:current_syntax")
+    finish
+endif
+
+syn include @python syntax/python.vim
+if exists("b:current_syntax")
+  unlet b:current_syntax
+endif
+
+" BitBake syntax
+
+" Matching case
+syn case match
+
+" Indicates the error when nothing is matched
+syn match bbUnmatched           "."
+
+" Comments
+syn cluster bbCommentGroup      contains=bbTodo,@Spell
+syn keyword bbTodo              COMBAK FIXME TODO XXX contained
+syn match bbComment             "#.*$" contains=@bbCommentGroup
+
+" String helpers
+syn match bbQuote               +['"]+ contained 
+syn match bbDelimiter           "[(){}=]" contained
+syn match bbArrayBrackets       "[\[\]]" contained
+
+" BitBake strings
+syn match bbContinue            "\\$"
+syn region bbString             matchgroup=bbQuote start=+"+ skip=+\\$+ end=+"+ contained contains=bbTodo,bbContinue,bbVarDeref,bbVarPyValue,@Spell
+syn region bbString             matchgroup=bbQuote start=+'+ skip=+\\$+ end=+'+ contained contains=bbTodo,bbContinue,bbVarDeref,bbVarPyValue,@Spell
+
+" Vars definition
+syn match bbExport            "^export" nextgroup=bbIdentifier skipwhite
+syn keyword bbExportFlag        export contained nextgroup=bbIdentifier skipwhite
+syn match bbIdentifier          "[a-zA-Z0-9\-_\.\/\+]\+" display contained
+syn match bbVarDeref            "${[a-zA-Z0-9\-_\.\/\+]\+}" contained
+syn match bbVarEq               "\(:=\|+=\|=+\|\.=\|=\.\|?=\|??=\|=\)" contained nextgroup=bbVarValue
+syn match bbVarDef              "^\(export\s*\)\?\([a-zA-Z0-9\-_\.\/\+]\+\(_[${}a-zA-Z0-9\-_\.\/\+]\+\)\?\)\s*\(:=\|+=\|=+\|\.=\|=\.\|?=\|??=\|=\)\@=" contains=bbExportFlag,bbIdentifier,bbVarDeref nextgroup=bbVarEq
+syn match bbVarValue            ".*$" contained contains=bbString,bbVarDeref,bbVarPyValue
+syn region bbVarPyValue         start=+${@+ skip=+\\$+ end=+}+ contained contains=@python
+
+" Vars metadata flags
+syn match bbVarFlagDef          "^\([a-zA-Z0-9\-_\.]\+\)\(\[[a-zA-Z0-9\-_\.]\+\]\)\@=" contains=bbIdentifier nextgroup=bbVarFlagFlag
+syn region bbVarFlagFlag        matchgroup=bbArrayBrackets start="\[" end="\]\s*\(=\|+=\|=+\|?=\)\@=" contained contains=bbIdentifier nextgroup=bbVarEq
+
+" Includes and requires
+syn keyword bbInclude           inherit include require contained 
+syn match bbIncludeRest         ".*$" contained contains=bbString,bbVarDeref
+syn match bbIncludeLine         "^\(inherit\|include\|require\)\s\+" contains=bbInclude nextgroup=bbIncludeRest
+
+" Add taks and similar
+syn keyword bbStatement         addtask addhandler after before EXPORT_FUNCTIONS contained
+syn match bbStatementRest       ".*$" skipwhite contained contains=bbStatement
+syn match bbStatementLine       "^\(addtask\|addhandler\|after\|before\|EXPORT_FUNCTIONS\)\s\+" contains=bbStatement nextgroup=bbStatementRest
+
+" OE Important Functions
+syn keyword bbOEFunctions       do_fetch do_unpack do_patch do_configure do_compile do_stage do_install do_package contained
+
+" Generic Functions
+syn match bbFunction            "\h[0-9A-Za-z_-]*" display contained contains=bbOEFunctions
+
+" BitBake shell metadata
+syn include @shell syntax/sh.vim
+if exists("b:current_syntax")
+  unlet b:current_syntax
+endif
+syn keyword bbShFakeRootFlag    fakeroot contained
+syn match bbShFuncDef           "^\(fakeroot\s*\)\?\([0-9A-Za-z_${}-]\+\)\(python\)\@<!\(\s*()\s*\)\({\)\@=" contains=bbShFakeRootFlag,bbFunction,bbVarDeref,bbDelimiter nextgroup=bbShFuncRegion skipwhite
+syn region bbShFuncRegion       matchgroup=bbDelimiter start="{\s*$" end="^}\s*$" contained contains=@shell
+
+" Python value inside shell functions
+syn region shDeref         start=+${@+ skip=+\\$+ excludenl end=+}+ contained contains=@python
+
+" BitBake python metadata
+syn keyword bbPyFlag            python contained
+syn match bbPyFuncDef           "^\(python\s\+\)\([0-9A-Za-z_${}-]\+\)\?\(\s*()\s*\)\({\)\@=" contains=bbPyFlag,bbFunction,bbVarDeref,bbDelimiter nextgroup=bbPyFuncRegion skipwhite
+syn region bbPyFuncRegion       matchgroup=bbDelimiter start="{\s*$" end="^}\s*$" contained contains=@python
+
+" BitBake 'def'd python functions
+syn keyword bbPyDef             def contained
+syn region bbPyDefRegion        start='^\(def\s\+\)\([0-9A-Za-z_-]\+\)\(\s*(.*)\s*\):\s*$' end='^\(\s\|$\)\@!' contains=@python
+
+" Highlighting Definitions
+hi def link bbUnmatched         Error
+hi def link bbInclude           Include
+hi def link bbTodo              Todo
+hi def link bbComment           Comment
+hi def link bbQuote             String
+hi def link bbString            String
+hi def link bbDelimiter         Keyword
+hi def link bbArrayBrackets     Statement
+hi def link bbContinue          Special
+hi def link bbExport            Type
+hi def link bbExportFlag        Type
+hi def link bbIdentifier	    Identifier
+hi def link bbVarDeref          PreProc
+hi def link bbVarDef            Identifier
+hi def link bbVarValue          String
+hi def link bbShFakeRootFlag    Type
+hi def link bbFunction          Function
+hi def link bbPyFlag            Type
+hi def link bbPyDef             Statement
+hi def link bbStatement         Statement
+hi def link bbStatementRest     Identifier
+hi def link bbOEFunctions       Special
+hi def link bbVarPyValue        PreProc
+
+let b:current_syntax = "bb"

bitbake/doc/COPYING.GPL

@@ -0,0 +1,339 @@
+		    GNU GENERAL PUBLIC LICENSE
+		       Version 2, June 1991
+
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
+ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+			    Preamble
+
+  The licenses for most software are designed to take away your
+freedom to share and change it.  By contrast, the GNU General Public
+License is intended to guarantee your freedom to share and change free
+software--to make sure the software is free for all its users.  This
+General Public License applies to most of the Free Software
+Foundation's software and to any other program whose authors commit to
+using it.  (Some other Free Software Foundation software is covered by
+the GNU Lesser General Public License instead.)  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get it
+if you want it, that you can change the software or use pieces of it
+in new free programs; and that you know you can do these things.
+
+  To protect your rights, we need to make restrictions that forbid
+anyone to deny you these rights or to ask you to surrender the rights.
+These restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must give the recipients all the rights that
+you have.  You must make sure that they, too, receive or can get the
+source code.  And you must show them these terms so they know their
+rights.
+
+  We protect your rights with two steps: (1) copyright the software, and
+(2) offer you this license which gives you legal permission to copy,
+distribute and/or modify the software.
+
+  Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software.  If the software is modified by someone else and passed on, we
+want its recipients to know that what they have is not the original, so
+that any problems introduced by others will not reflect on the original
+authors' reputations.
+
+  Finally, any free program is threatened constantly by software
+patents.  We wish to avoid the danger that redistributors of a free
+program will individually obtain patent licenses, in effect making the
+program proprietary.  To prevent this, we have made it clear that any
+patent must be licensed for everyone's free use or not licensed at all.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+		    GNU GENERAL PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. This License applies to any program or other work which contains
+a notice placed by the copyright holder saying it may be distributed
+under the terms of this General Public License.  The "Program", below,
+refers to any such program or work, and a "work based on the Program"
+means either the Program or any derivative work under copyright law:
+that is to say, a work containing the Program or a portion of it,
+either verbatim or with modifications and/or translated into another
+language.  (Hereinafter, translation is included without limitation in
+the term "modification".)  Each licensee is addressed as "you".
+
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope.  The act of
+running the Program is not restricted, and the output from the Program
+is covered only if its contents constitute a work based on the
+Program (independent of having been made by running the Program).
+Whether that is true depends on what the Program does.
+
+  1. You may copy and distribute verbatim copies of the Program's
+source code as you receive it, in any medium, provided that you
+conspicuously and appropriately publish on each copy an appropriate
+copyright notice and disclaimer of warranty; keep intact all the
+notices that refer to this License and to the absence of any warranty;
+and give any other recipients of the Program a copy of this License
+along with the Program.
+
+You may charge a fee for the physical act of transferring a copy, and
+you may at your option offer warranty protection in exchange for a fee.
+
+  2. You may modify your copy or copies of the Program or any portion
+of it, thus forming a work based on the Program, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+    a) You must cause the modified files to carry prominent notices
+    stating that you changed the files and the date of any change.
+
+    b) You must cause any work that you distribute or publish, that in
+    whole or in part contains or is derived from the Program or any
+    part thereof, to be licensed as a whole at no charge to all third
+    parties under the terms of this License.
+
+    c) If the modified program normally reads commands interactively
+    when run, you must cause it, when started running for such
+    interactive use in the most ordinary way, to print or display an
+    announcement including an appropriate copyright notice and a
+    notice that there is no warranty (or else, saying that you provide
+    a warranty) and that users may redistribute the program under
+    these conditions, and telling the user how to view a copy of this
+    License.  (Exception: if the Program itself is interactive but
+    does not normally print such an announcement, your work based on
+    the Program is not required to print an announcement.)
+
+These requirements apply to the modified work as a whole.  If
+identifiable sections of that work are not derived from the Program,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works.  But when you
+distribute the same sections as part of a whole which is a work based
+on the Program, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+  3. You may copy and distribute the Program (or a work based on it,
+under Section 2) in object code or executable form under the terms of
+Sections 1 and 2 above provided that you also do one of the following:
+
+    a) Accompany it with the complete corresponding machine-readable
+    source code, which must be distributed under the terms of Sections
+    1 and 2 above on a medium customarily used for software interchange; or,
+
+    b) Accompany it with a written offer, valid for at least three
+    years, to give any third party, for a charge no more than your
+    cost of physically performing source distribution, a complete
+    machine-readable copy of the corresponding source code, to be
+    distributed under the terms of Sections 1 and 2 above on a medium
+    customarily used for software interchange; or,
+
+    c) Accompany it with the information you received as to the offer
+    to distribute corresponding source code.  (This alternative is
+    allowed only for noncommercial distribution and only if you
+    received the program in object code or executable form with such
+    an offer, in accord with Subsection b above.)
+
+The source code for a work means the preferred form of the work for
+making modifications to it.  For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to
+control compilation and installation of the executable.  However, as a
+special exception, the source code distributed need not include
+anything that is normally distributed (in either source or binary
+form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component
+itself accompanies the executable.
+
+If distribution of executable or object code is made by offering
+access to copy from a designated place, then offering equivalent
+access to copy the source code from the same place counts as
+distribution of the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+  4. You may not copy, modify, sublicense, or distribute the Program
+except as expressly provided under this License.  Any attempt
+otherwise to copy, modify, sublicense or distribute the Program is
+void, and will automatically terminate your rights under this License.
+However, parties who have received copies, or rights, from you under
+this License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+  5. You are not required to accept this License, since you have not
+signed it.  However, nothing else grants you permission to modify or
+distribute the Program or its derivative works.  These actions are
+prohibited by law if you do not accept this License.  Therefore, by
+modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Program or works based on it.
+
+  6. Each time you redistribute the Program (or any work based on the
+Program), the recipient automatically receives a license from the
+original licensor to copy, distribute or modify the Program subject to
+these terms and conditions.  You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties to
+this License.
+
+  7. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Program at all.  For example, if a patent
+license would not permit royalty-free redistribution of the Program by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system, which is
+implemented by public license practices.  Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+  8. If the distribution and/or use of the Program is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Program under this License
+may add an explicit geographical distribution limitation excluding
+those countries, so that distribution is permitted only in or among
+countries not thus excluded.  In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+  9. The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+Each version is given a distinguishing version number.  If the Program
+specifies a version number of this License which applies to it and "any
+later version", you have the option of following the terms and conditions
+either of that version or of any later version published by the Free
+Software Foundation.  If the Program does not specify a version number of
+this License, you may choose any version ever published by the Free Software
+Foundation.
+
+  10. If you wish to incorporate parts of the Program into other free
+programs whose distribution conditions are different, write to the author
+to ask for permission.  For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this.  Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software and
+of promoting the sharing and reuse of software generally.
+
+			    NO WARRANTY
+
+  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+		     END OF TERMS AND CONDITIONS
+
+	    How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+Also add information on how to contact you by electronic and paper mail.
+
+If the program is interactive, make it output a short notice like this
+when it starts in an interactive mode:
+
+    Gnomovision version 69, Copyright (C) year name of author
+    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, the commands you use may
+be called something other than `show w' and `show c'; they could even be
+mouse-clicks or menu items--whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary.  Here is a sample; alter the names:
+
+  Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+  `Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+  <signature of Ty Coon>, 1 April 1989
+  Ty Coon, President of Vice
+
+This General Public License does not permit incorporating your program into
+proprietary programs.  If your program is a subroutine library, you may
+consider it more useful to permit linking proprietary applications with the
+library.  If this is what you want to do, use the GNU Lesser General
+Public License instead of this License.

bitbake/doc/COPYING.MIT

@@ -0,0 +1,17 @@
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
+SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR
+THE USE OR OTHER DEALINGS IN THE SOFTWARE.

bitbake/doc/Makefile

@@ -0,0 +1,91 @@
+# This is a single Makefile to handle all generated BitBake documents.
+# The Makefile needs to live in the documentation directory and all figures used
+# in any manuals must be .PNG files and live in the individual book's figures
+# directory.
+#
+# The Makefile has these targets:
+#
+#    pdf:      generates a PDF version of a manual.
+#    html:     generates an HTML version of a manual.
+#    tarball:  creates a tarball for the doc files.
+#    validate: validates
+#    clean:    removes files
+#
+# The Makefile generates an HTML version of every document.  The
+# variable DOC indicates the folder name for a given manual.
+#
+# To build a manual, you must invoke 'make' with the DOC argument.
+#
+# Examples:
+#
+#     make DOC=bitbake-user-manual
+#     make pdf DOC=bitbake-user-manual
+#
+# The first example generates the HTML version of the User Manual.
+# The second example generates the PDF version of the User Manual.
+#
+
+ifeq ($(DOC),bitbake-user-manual)
+XSLTOPTS = --stringparam html.stylesheet bitbake-user-manual-style.css \
+           --stringparam  chapter.autolabel 1 \
+           --stringparam  section.autolabel 1 \
+           --stringparam  section.label.includes.component.label 1 \
+           --xinclude
+ALLPREQ = html tarball
+TARFILES = bitbake-user-manual-style.css bitbake-user-manual.html figures/bitbake-title.png
+MANUALS = $(DOC)/$(DOC).html
+FIGURES = figures
+STYLESHEET = $(DOC)/*.css
+
+endif
+
+##
+# These URI should be rewritten by your distribution's xml catalog to
+# match your localy installed XSL stylesheets.
+XSL_BASE_URI  = http://docbook.sourceforge.net/release/xsl/current
+XSL_XHTML_URI = $(XSL_BASE_URI)/xhtml/docbook.xsl
+
+all: $(ALLPREQ)
+
+pdf:
+ifeq ($(DOC),bitbake-user-manual)
+	@echo " "
+	@echo "********** Building."$(DOC)
+	@echo " "
+	cd $(DOC); ../tools/docbook-to-pdf $(DOC).xml ../template; cd ..
+endif
+
+html:
+ifeq ($(DOC),bitbake-user-manual)
+#       See http://www.sagehill.net/docbookxsl/HtmlOutput.html
+	@echo " "
+	@echo "******** Building "$(DOC)
+	@echo " "
+	cd $(DOC); xsltproc $(XSLTOPTS) -o $(DOC).html $(DOC)-customization.xsl $(DOC).xml; cd ..
+endif
+
+tarball: html
+	@echo " "
+	@echo "******** Creating Tarball of document files"
+	@echo " "
+	cd $(DOC); tar -cvzf $(DOC).tgz $(TARFILES); cd ..
+
+validate:
+	cd $(DOC); xmllint --postvalid --xinclude --noout $(DOC).xml; cd ..
+
+publish:
+	@if test -f $(DOC)/$(DOC).html; \
+	  then \
+            echo " "; \
+            echo "******** Publishing "$(DOC)".html"; \
+            echo " "; \
+            scp -r $(MANUALS) $(STYLESHEET) docs.yp:/var/www/www.yoctoproject.org-docs/$(VER)/$(DOC); \
+            cd $(DOC); scp -r $(FIGURES) docs.yp:/var/www/www.yoctoproject.org-docs/$(VER)/$(DOC); \
+	else \
+          echo " "; \
+          echo $(DOC)".html missing.  Generate the file first then try again."; \
+          echo " "; \
+	fi
+
+clean:
+	rm -rf $(MANUALS); rm $(DOC)/$(DOC).tgz;

bitbake/doc/README

@@ -0,0 +1,39 @@
+Documentation
+=============
+
+This is the directory that contains the BitBake documentation. 
+
+Manual Organization
+===================
+
+Folders exist for individual manuals as follows:
+
+* bitbake-user-manual      - The BitBake User Manual 
+
+Each folder is self-contained regarding content and figures.
+
+If you want to find HTML versions of the BitBake manuals on the web, 
+go to http://www.openembedded.org/wiki/Documentation. 
+
+Makefile
+========
+
+The Makefile processes manual directories to create HTML, PDF,
+tarballs, etc.  Details on how the Makefile work are documented
+inside the Makefile.  See that file for more information.
+
+To build a manual, you run the make command and pass it the name
+of the folder containing the manual's contents. 
+For example, the following command run from the documentation directory 
+creates an HTML and a PDF version of the BitBake User Manual.
+The DOC variable specifies the manual you are making:
+
+     $ make DOC=bitbake-user-manual
+
+template
+========
+Contains various templates, fonts, and some old PNG files.
+
+tools
+=====
+Contains a tool to convert the DocBook files to PDF format.

bitbake/doc/bitbake-user-manual/bitbake-user-manual-customization.xsl

@@ -0,0 +1,29 @@
+<?xml version='1.0'?>
+<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" xmlns:fo="http://www.w3.org/1999/XSL/Format" version="1.0">
+
+  <xsl:import href="http://downloads.yoctoproject.org/mirror/docbook-mirror/docbook-xsl-1.76.1/xhtml/docbook.xsl" />
+
+<!--
+
+  <xsl:import href="../template/1.76.1/docbook-xsl-1.76.1/xhtml/docbook.xsl" />
+
+  <xsl:import href="http://docbook.sourceforge.net/release/xsl/1.76.1/xhtml/docbook.xsl" />
+
+-->
+
+  <xsl:include href="../template/permalinks.xsl"/>
+  <xsl:include href="../template/section.title.xsl"/>
+  <xsl:include href="../template/component.title.xsl"/>
+  <xsl:include href="../template/division.title.xsl"/>
+  <xsl:include href="../template/formal.object.heading.xsl"/>
+  <xsl:include href="../template/gloss-permalinks.xsl"/>
+
+  <xsl:param name="html.stylesheet" select="'user-manual-style.css'" />
+  <xsl:param name="chapter.autolabel" select="1" />
+  <xsl:param name="section.autolabel" select="1" />
+  <xsl:param name="section.label.includes.component.label" select="1" />
+  <xsl:param name="appendix.autolabel">A</xsl:param>
+
+<!--  <xsl:param name="generate.toc" select="'article nop'"></xsl:param>  -->
+
+</xsl:stylesheet>

bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml

@@ -0,0 +1,931 @@
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+
+<chapter id="bitbake-user-manual-execution">
+    <title>Execution</title>
+
+    <para>
+        The primary purpose for running BitBake is to produce some kind
+        of output such as a single installable package, a kernel, a software
+        development kit, or even a full, board-specific bootable Linux image,
+        complete with bootloader, kernel, and root filesystem.
+        Of course, you can execute the <filename>bitbake</filename>
+        command with options that cause it to execute single tasks,
+        compile single recipe files, capture or clear data, or simply
+        return information about the execution environment.
+    </para>
+
+    <para>
+        This chapter describes BitBake's execution process from start
+        to finish when you use it to create an image.
+        The execution process is launched using the following command
+        form:
+        <literallayout class='monospaced'>
+     $ bitbake <replaceable>target</replaceable>
+        </literallayout>
+        For information on the BitBake command and its options,
+        see
+        "<link linkend='bitbake-user-manual-command'>The BitBake Command</link>"
+        section.
+        <note>
+            <para>
+                Prior to executing BitBake, you should take advantage of available
+                parallel thread execution on your build host by setting the
+                <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
+                variable in your project's <filename>local.conf</filename>
+                configuration file.
+            </para>
+
+            <para>
+                A common method to determine this value for your build host is to run
+                the following:
+                <literallayout class='monospaced'>
+     $ grep processor /proc/cpuinfo
+                </literallayout>
+                This command returns the number of processors, which takes into
+                account hyper-threading.
+                Thus, a quad-core build host with hyper-threading most likely
+                shows eight processors, which is the value you would then assign to
+                <filename>BB_NUMBER_THREADS</filename>.
+            </para>
+
+            <para>
+                A possibly simpler solution is that some Linux distributions
+                (e.g. Debian and Ubuntu) provide the <filename>ncpus</filename> command.
+            </para>
+        </note>
+    </para>
+
+    <section id='parsing-the-base-configuration-metadata'>
+        <title>Parsing the Base Configuration Metadata</title>
+
+        <para>
+            The first thing BitBake does is parse base configuration
+            metadata.
+            Base configuration metadata consists of your project's
+            <filename>bblayers.conf</filename> file to determine what
+            layers BitBake needs to recognize, all necessary
+            <filename>layer.conf</filename> files (one from each layer),
+            and <filename>bitbake.conf</filename>.
+            The data itself is of various types:
+            <itemizedlist>
+                <listitem><para><emphasis>Recipes:</emphasis>
+                    Details about particular pieces of software.
+                    </para></listitem>
+                <listitem><para><emphasis>Class Data:</emphasis>
+                    An abstraction of common build information
+                    (e.g. how to build a Linux kernel).
+                    </para></listitem>
+                <listitem><para><emphasis>Configuration Data:</emphasis>
+                    Machine-specific settings, policy decisions,
+                    and so forth.
+                    Configuration data acts as the glue to bind everything
+                    together.</para></listitem>
+            </itemizedlist>
+        </para>
+
+        <para>
+            The <filename>layer.conf</filename> files are used to
+            construct key variables such as
+            <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
+            and
+            <link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
+            <filename>BBPATH</filename> is used to search for
+            configuration and class files under the
+            <filename>conf</filename> and <filename>classes</filename>
+            directories, respectively.
+            <filename>BBFILES</filename> is used to locate both recipe
+            and recipe append files
+            (<filename>.bb</filename> and <filename>.bbappend</filename>).
+            If there is no <filename>bblayers.conf</filename> file,
+            it is assumed the user has set the <filename>BBPATH</filename>
+            and <filename>BBFILES</filename> directly in the environment.
+        </para>
+
+        <para>
+            Next, the <filename>bitbake.conf</filename> file is located
+            using the <filename>BBPATH</filename> variable that was
+            just constructed.
+            The <filename>bitbake.conf</filename> file may also include other
+            configuration files using the
+            <filename>include</filename> or
+            <filename>require</filename> directives.
+        </para>
+
+        <para>
+            Prior to parsing configuration files, Bitbake looks
+            at certain variables, including:
+            <itemizedlist>
+                <listitem><para>
+                    <link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
+                    </para></listitem>
+                <listitem><para>
+                    <link linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
+                    </para></listitem>
+                <listitem><para>
+                    <link linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
+                    </para></listitem>
+                <listitem><para>
+                    <link linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
+                    </para></listitem>
+                <listitem><para>
+                    <link linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
+                    </para></listitem>
+            </itemizedlist>
+            The first four variables in this list relate to how BitBake treats shell
+            environment variables during task execution.
+            By default, BitBake cleans the environment variables and provides tight
+            control over the shell execution environment.
+            However, through the use of these first four variables, you can
+            apply your control regarding the
+            environment variables allowed to be used by BitBake in the shell
+            during execution of tasks.
+            See the
+            "<link linkend='passing-information-into-the-build-task-environment'>Passing Information Into the Build Task Environment</link>"
+            section and the information about these variables in the
+            variable glossary for more information on how they work and
+            on how to use them.
+        </para>
+
+        <para>
+            The base configuration metadata is global
+            and therefore affects all recipes and tasks that are executed.
+        </para>
+
+        <para>
+            BitBake first searches the current working directory for an
+            optional <filename>conf/bblayers.conf</filename> configuration file.
+            This file is expected to contain a
+            <link linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link>
+            variable that is a space-delimited list of 'layer' directories.
+            Recall that if BitBake cannot find a <filename>bblayers.conf</filename>
+            file, then it is assumed the user has set the <filename>BBPATH</filename>
+            and <filename>BBFILES</filename> variables directly in the environment.
+        </para>
+
+        <para>
+            For each directory (layer) in this list, a <filename>conf/layer.conf</filename>
+            file is located and parsed with the
+            <link linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link>
+            variable being set to the directory where the layer was found.
+            The idea is these files automatically set up
+            <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
+            and other variables correctly for a given build directory.
+        </para>
+
+        <para>
+            BitBake then expects to find the <filename>conf/bitbake.conf</filename>
+            file somewhere in the user-specified <filename>BBPATH</filename>.
+            That configuration file generally has include directives to pull
+            in any other metadata such as files specific to the architecture,
+            the machine, the local environment, and so forth.
+        </para>
+
+        <para>
+            Only variable definitions and include directives are allowed
+            in BitBake <filename>.conf</filename> files.
+            Some variables directly influence BitBake's behavior.
+            These variables might have been set from the environment
+            depending on the environment variables previously
+            mentioned or set in the configuration files.
+            The
+            "<link linkend='ref-variables-glos'>Variables Glossary</link>"
+            chapter presents a full list of variables.
+        </para>
+
+        <para>
+            After parsing configuration files, BitBake uses its rudimentary
+            inheritance mechanism, which is through class files, to inherit
+            some standard classes.
+            BitBake parses a class when the inherit directive responsible
+            for getting that class is encountered.
+        </para>
+
+        <para>
+            The <filename>base.bbclass</filename> file is always included.
+            Other classes that are specified in the configuration using the
+            <link linkend='var-INHERIT'><filename>INHERIT</filename></link>
+            variable are also included.
+            BitBake searches for class files in a
+            <filename>classes</filename> subdirectory under
+            the paths in <filename>BBPATH</filename> in the same way as
+            configuration files.
+        </para>
+
+        <para>
+            A good way to get an idea of the configuration files and
+            the class files used in your execution environment is to
+            run the following BitBake command:
+            <literallayout class='monospaced'>
+     $ bitbake -e > mybb.log
+            </literallayout>
+            Examining the top of the <filename>mybb.log</filename>
+            shows you the many configuration files and class files
+            used in your execution environment.
+        </para>
+
+        <note>
+            <para>
+                You need to be aware of how BitBake parses curly braces.
+                If a recipe uses a closing curly brace within the function and
+                the character has no leading spaces, BitBake produces a parsing
+                error.
+                If you use a pair of curly braces in a shell function, the
+                closing curly brace must not be located at the start of the line
+                without leading spaces.
+            </para>
+
+            <para>
+                Here is an example that causes BitBake to produce a parsing
+                error:
+                <literallayout class='monospaced'>
+     fakeroot create_shar() {
+         cat &lt;&lt; "EOF" &gt; ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+     usage()
+     {
+       echo "test"
+       ###### The following "}" at the start of the line causes a parsing error ######
+     }
+     EOF
+     }
+                </literallayout>
+                Writing the recipe this way avoids the error:
+                <literallayout class='monospaced'>
+     fakeroot create_shar() {
+         cat &lt;&lt; "EOF" &gt; ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh
+     usage()
+     {
+       echo "test"
+       ######The following "}" with a leading space at the start of the line avoids the error ######
+      }
+     EOF
+     }
+                </literallayout>
+            </para>
+        </note>
+    </section>
+
+    <section id='locating-and-parsing-recipes'>
+        <title>Locating and Parsing Recipes</title>
+
+        <para>
+            During the configuration phase, BitBake will have set
+            <link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
+            BitBake now uses it to construct a list of recipes to parse,
+            along with any append files (<filename>.bbappend</filename>)
+            to apply.
+            <filename>BBFILES</filename> is a space-separated list of
+            available files and supports wildcards.
+            An example would be:
+            <literallayout class='monospaced'>
+     BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend"
+            </literallayout>
+            BitBake parses each recipe and append file located
+            with <filename>BBFILES</filename> and stores the values of
+            various variables into the datastore.
+            <note>
+                Append files are applied in the order they are encountered in
+                <filename>BBFILES</filename>.
+            </note>
+            For each file, a fresh copy of the base configuration is
+            made, then the recipe is parsed line by line.
+            Any inherit statements cause BitBake to find and
+            then parse class files (<filename>.bbclass</filename>)
+            using
+            <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
+            as the search path.
+            Finally, BitBake parses in order any append files found in
+            <filename>BBFILES</filename>.
+        </para>
+
+        <para>
+            One common convention is to use the recipe filename to define
+            pieces of metadata.
+            For example, in <filename>bitbake.conf</filename> the recipe
+            name and version are used to set the variables
+            <link linkend='var-PN'><filename>PN</filename></link> and
+            <link linkend='var-PV'><filename>PV</filename></link>:
+            <literallayout class='monospaced'>
+     PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
+     PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
+            </literallayout>
+            In this example, a recipe called "something_1.2.3.bb" would set
+            <filename>PN</filename> to "something" and
+            <filename>PV</filename> to "1.2.3".
+        </para>
+
+        <para>
+            By the time parsing is complete for a recipe, BitBake
+            has a list of tasks that the recipe defines and a set of
+            data consisting of keys and values as well as
+            dependency information about the tasks.
+        </para>
+
+        <para>
+            BitBake does not need all of this information.
+            It only needs a small subset of the information to make
+            decisions about the recipe.
+            Consequently, BitBake caches the values in which it is
+            interested and does not store the rest of the information.
+            Experience has shown it is faster to re-parse the metadata than to
+            try and write it out to the disk and then reload it.
+        </para>
+
+        <para>
+            Where possible, subsequent BitBake commands reuse this cache of
+            recipe information.
+            The validity of this cache is determined by first computing a
+            checksum of the base configuration data (see
+            <link linkend='var-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
+            and then checking if the checksum matches.
+            If that checksum matches what is in the cache and the recipe
+            and class files have not changed, Bitbake is able to use
+            the cache.
+            BitBake then reloads the cached information about the recipe
+            instead of reparsing it from scratch.
+        </para>
+
+        <para>
+            Recipe file collections exist to allow the user to
+            have multiple repositories of
+            <filename>.bb</filename> files that contain the same
+            exact package.
+            For example, one could easily use them to make one's
+            own local copy of an upstream repository, but with
+            custom modifications that one does not want upstream.
+            Here is an example:
+            <literallayout class='monospaced'>
+    BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb"
+    BBFILE_COLLECTIONS = "upstream local"
+    BBFILE_PATTERN_upstream = "^/stuff/openembedded/"
+    BBFILE_PATTERN_local = "^/stuff/openembedded.modified/"
+    BBFILE_PRIORITY_upstream = "5"
+    BBFILE_PRIORITY_local = "10"
+            </literallayout>
+            <note>
+                The layers mechanism is now the preferred method of collecting
+                code.
+                While the collections code remains, its main use is to set layer
+                priorities and to deal with overlap (conflicts) between layers.
+            </note>
+        </para>
+    </section>
+
+    <section id='bb-bitbake-providers'>
+        <title>Providers</title>
+
+        <para>
+            Assuming BitBake has been instructed to execute a target
+            and that all the recipe files have been parsed, BitBake
+            starts to figure out how to build the target.
+            BitBake looks through the <filename>PROVIDES</filename> list
+            for each of the recipes.
+            A <filename>PROVIDES</filename> list is the list of names by which
+            the recipe can be known.
+            Each recipe's <filename>PROVIDES</filename> list is created
+            implicitly through the recipe's
+            <link linkend='var-PN'><filename>PN</filename></link> variable
+            and explicitly through the recipe's
+            <link linkend='var-PROVIDES'><filename>PROVIDES</filename></link>
+            variable, which is optional.
+        </para>
+
+        <para>
+            When a recipe uses <filename>PROVIDES</filename>, that recipe's
+            functionality can be found under an alternative name or names other
+            than the implicit <filename>PN</filename> name.
+            As an example, suppose a recipe named <filename>keyboard_1.0.bb</filename>
+            contained the following:
+            <literallayout class='monospaced'>
+     PROVIDES += "fullkeyboard"
+            </literallayout>
+            The <filename>PROVIDES</filename> list for this recipe becomes
+            "keyboard", which is implicit, and "fullkeyboard", which is explicit.
+            Consequently, the functionality found in
+            <filename>keyboard_1.0.bb</filename> can be found under two
+            different names.
+        </para>
+    </section>
+
+    <section id='bb-bitbake-preferences'>
+        <title>Preferences</title>
+
+        <para>
+            The <filename>PROVIDES</filename> list is only part of the solution
+            for figuring out a target's recipes.
+            Because targets might have multiple providers, BitBake needs
+            to prioritize providers by determining provider preferences.
+        </para>
+
+        <para>
+            A common example in which a target has multiple providers
+            is "virtual/kernel", which is on the
+            <filename>PROVIDES</filename> list for each kernel recipe.
+            Each machine often selects the best kernel provider by using a
+            line similar to the following in the machine configuration file:
+            <literallayout class='monospaced'>
+     PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
+            </literallayout>
+            The default
+            <link linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
+            is the provider with the same name as the target.
+            Bitbake iterates through each target it needs to build and
+            resolves them and their dependencies using this process.
+        </para>
+
+        <para>
+            Understanding how providers are chosen is made complicated by the fact
+            that multiple versions might exist for a given provider.
+            BitBake defaults to the highest version of a provider.
+            Version comparisons are made using the same method as Debian.
+            You can use the
+            <link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
+            variable to specify a particular version.
+            You can influence the order by using the
+            <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
+            variable.
+        </para>
+
+        <para>
+            By default, files have a preference of "0".
+            Setting <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the
+            recipe unlikely to be used unless it is explicitly referenced.
+            Setting <filename>DEFAULT_PREFERENCE</filename> to "1" makes it
+            likely the recipe is used.
+            <filename>PREFERRED_VERSION</filename> overrides any
+            <filename>DEFAULT_PREFERENCE</filename> setting.
+            <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer
+            and more experimental recipe versions until they have undergone
+            sufficient testing to be considered stable.
+        </para>
+
+        <para>
+            When there are multiple “versions” of a given recipe,
+            BitBake defaults to selecting the most recent
+            version, unless otherwise specified.
+            If the recipe in question has a
+            <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
+            set lower than the other recipes (default is 0), then
+            it will not be selected.
+            This allows the person or persons maintaining
+            the repository of recipe files to specify
+            their preference for the default selected version.
+            Additionally, the user can specify their preferred version.
+        </para>
+
+        <para>
+            If the first recipe is named <filename>a_1.1.bb</filename>, then the
+            <link linkend='var-PN'><filename>PN</filename></link> variable
+            will be set to “a”, and the
+            <link linkend='var-PV'><filename>PV</filename></link>
+            variable will be set to 1.1.
+        </para>
+
+        <para>
+            Thus, if a recipe named <filename>a_1.2.bb</filename> exists, BitBake
+            will choose 1.2 by default.
+            However, if you define the following variable in a
+            <filename>.conf</filename> file that BitBake parses, you
+            can change that preference:
+            <literallayout class='monospaced'>
+     PREFERRED_VERSION_a = "1.1"
+            </literallayout>
+        </para>
+
+        <note>
+            <para>
+                It is common for a recipe to provide two versions -- a stable,
+                numbered (and preferred) version, and a version that is
+                automatically checked out from a source code repository that
+                is considered more "bleeding edge" but can be selected only
+                explicitly.
+            </para>
+
+            <para>
+                For example, in the OpenEmbedded codebase, there is a standard,
+                versioned recipe file for BusyBox,
+                <filename>busybox_1.22.1.bb</filename>,
+                but there is also a Git-based version,
+                <filename>busybox_git.bb</filename>, which explicitly contains the line
+                <literallayout class='monospaced'>
+    DEFAULT_PREFERENCE = "-1"
+                </literallayout>
+                to ensure that the numbered, stable version is always preferred
+                unless the developer selects otherwise.
+            </para>
+        </note>
+    </section>
+
+    <section id='bb-bitbake-dependencies'>
+        <title>Dependencies</title>
+
+        <para>
+            Each target BitBake builds consists of multiple tasks such as
+            <filename>fetch</filename>, <filename>unpack</filename>,
+            <filename>patch</filename>, <filename>configure</filename>,
+            and <filename>compile</filename>.
+            For best performance on multi-core systems, BitBake considers each
+            task as an independent
+            entity with its own set of dependencies.
+        </para>
+
+        <para>
+            Dependencies are defined through several variables.
+            You can find information about variables BitBake uses in
+            the <link linkend='ref-variables-glos'>Variables Glossary</link>
+            near the end of this manual.
+            At a basic level, it is sufficient to know that BitBake uses the
+            <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link> and
+            <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link> variables when
+            calculating dependencies.
+        </para>
+
+        <para>
+            For more information on how BitBake handles dependencies, see the
+            "<link linkend='dependencies'>Dependencies</link>" section.
+        </para>
+    </section>
+
+    <section id='ref-bitbake-tasklist'>
+        <title>The Task List</title>
+
+        <para>
+            Based on the generated list of providers and the dependency information,
+            BitBake can now calculate exactly what tasks it needs to run and in what
+            order it needs to run them.
+            The
+            "<link linkend='executing-tasks'>Executing Tasks</link>" section has more
+            information on how BitBake chooses which task to execute next.
+        </para>
+
+        <para>
+            The build now starts with BitBake forking off threads up to the limit set in the
+            <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
+            variable.
+            BitBake continues to fork threads as long as there are tasks ready to run,
+            those tasks have all their dependencies met, and the thread threshold has not been
+            exceeded.
+        </para>
+
+        <para>
+            It is worth noting that you can greatly speed up the build time by properly setting
+            the <filename>BB_NUMBER_THREADS</filename> variable.
+        </para>
+
+        <para>
+            As each task completes, a timestamp is written to the directory specified by the
+            <link linkend='var-STAMP'><filename>STAMP</filename></link> variable.
+            On subsequent runs, BitBake looks in the build directory within
+            <filename>tmp/stamps</filename> and does not rerun
+            tasks that are already completed unless a timestamp is found to be invalid.
+            Currently, invalid timestamps are only considered on a per
+            recipe file basis.
+            So, for example, if the configure stamp has a timestamp greater than the
+            compile timestamp for a given target, then the compile task would rerun.
+            Running the compile task again, however, has no effect on other providers
+            that depend on that target.
+        </para>
+
+        <para>
+            The exact format of the stamps is partly configurable.
+            In modern versions of BitBake, a hash is appended to the
+            stamp so that if the configuration changes, the stamp becomes
+            invalid and the task is automatically rerun.
+            This hash, or signature used, is governed by the signature policy
+            that is configured (see the
+            "<link linkend='checksums'>Checksums (Signatures)</link>"
+            section for information).
+            It is also possible to append extra metadata to the stamp using
+            the "stamp-extra-info" task flag.
+            For example, OpenEmbedded uses this flag to make some tasks machine-specific.
+        </para>
+
+        <note>
+            Some tasks are marked as "nostamp" tasks.
+            No timestamp file is created when these tasks are run.
+            Consequently, "nostamp" tasks are always rerun.
+        </note>
+
+        <para>
+            For more information on tasks, see the
+            "<link linkend='tasks'>Tasks</link>" section.
+        </para>
+    </section>
+
+    <section id='executing-tasks'>
+        <title>Executing Tasks</title>
+
+        <para>
+            Tasks can be either a shell task or a Python task.
+            For shell tasks, BitBake writes a shell script to
+            <filename>${</filename><link linkend='var-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename>
+            and then executes the script.
+            The generated shell script contains all the exported variables,
+            and the shell functions with all variables expanded.
+            Output from the shell script goes to the file
+            <filename>${T}/log.do_taskname.pid</filename>.
+            Looking at the expanded shell functions in the run file and
+            the output in the log files is a useful debugging technique.
+        </para>
+
+        <para>
+            For Python tasks, BitBake executes the task internally and logs
+            information to the controlling terminal.
+            Future versions of BitBake will write the functions to files
+            similar to the way shell tasks are handled.
+            Logging will be handled in a way similar to shell tasks as well.
+        </para>
+
+        <para>
+            The order in which BitBake runs the tasks is controlled by its
+            task scheduler.
+            It is possible to configure the scheduler and define custom
+            implementations for specific use cases.
+            For more information, see these variables that control the
+            behavior:
+            <itemizedlist>
+                <listitem><para>
+                    <link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
+                    </para></listitem>
+                <listitem><para>
+                    <link linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
+                    </para></listitem>
+            </itemizedlist>
+            It is possible to have functions run before and after a task's main
+            function.
+            This is done using the "prefuncs" and "postfuncs" flags of the task
+            that lists the functions to run.
+        </para>
+    </section>
+
+    <section id='checksums'>
+        <title>Checksums (Signatures)</title>
+
+        <para>
+            A checksum is a unique signature of a task's inputs.
+            The signature of a task can be used to determine if a task
+            needs to be run.
+            Because it is a change in a task's inputs that triggers running
+            the task, BitBake needs to detect all the inputs to a given task.
+            For shell tasks, this turns out to be fairly easy because
+            BitBake generates a "run" shell script for each task and
+            it is possible to create a checksum that gives you a good idea of when
+            the task's data changes.
+        </para>
+
+        <para>
+            To complicate the problem, some things should not be included in
+            the checksum.
+            First, there is the actual specific build path of a given task -
+            the working directory.
+            It does not matter if the working directory changes because it should not
+            affect the output for target packages.
+            The simplistic approach for excluding the working directory is to set
+            it to some fixed value and create the checksum for the "run" script.
+            BitBake goes one step better and uses the
+            <link linkend='var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
+            variable to define a list of variables that should never be included
+            when generating the signatures.
+        </para>
+
+        <para>
+            Another problem results from the "run" scripts containing functions that
+            might or might not get called.
+            The incremental build solution contains code that figures out dependencies
+            between shell functions.
+            This code is used to prune the "run" scripts down to the minimum set,
+            thereby alleviating this problem and making the "run" scripts much more
+            readable as a bonus.
+        </para>
+
+        <para>
+            So far we have solutions for shell scripts.
+            What about Python tasks?
+            The same approach applies even though these tasks are more difficult.
+            The process needs to figure out what variables a Python function accesses
+            and what functions it calls.
+            Again, the incremental build solution contains code that first figures out
+            the variable and function dependencies, and then creates a checksum for the data
+            used as the input to the task.
+        </para>
+
+        <para>
+            Like the working directory case, situations exist where dependencies
+            should be ignored.
+            For these cases, you can instruct the build process to ignore a dependency
+            by using a line like the following:
+            <literallayout class='monospaced'>
+     PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
+            </literallayout>
+            This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not
+            depend on the value of <filename>MACHINE</filename>, even if it does reference it.
+        </para>
+
+        <para>
+            Equally, there are cases where we need to add dependencies BitBake
+            is not able to find.
+            You can accomplish this by using a line like the following:
+            <literallayout class='monospaced'>
+      PACKAGE_ARCHS[vardeps] = "MACHINE"
+            </literallayout>
+            This example explicitly adds the <filename>MACHINE</filename> variable as a
+            dependency for <filename>PACKAGE_ARCHS</filename>.
+        </para>
+
+        <para>
+            Consider a case with in-line Python, for example, where BitBake is not
+            able to figure out dependencies.
+            When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
+            produces output when it discovers something for which it cannot figure out
+            dependencies.
+        </para>
+
+        <para>
+            Thus far, this section has limited discussion to the direct inputs into a task.
+            Information based on direct inputs is referred to as the "basehash" in the
+            code.
+            However, there is still the question of a task's indirect inputs - the
+            things that were already built and present in the build directory.
+            The checksum (or signature) for a particular task needs to add the hashes
+            of all the tasks on which the particular task depends.
+            Choosing which dependencies to add is a policy decision.
+            However, the effect is to generate a master checksum that combines the basehash
+            and the hashes of the task's dependencies.
+        </para>
+
+        <para>
+            At the code level, there are a variety of ways both the basehash and the
+            dependent task hashes can be influenced.
+            Within the BitBake configuration file, we can give BitBake some extra information
+            to help it construct the basehash.
+            The following statement effectively results in a list of global variable
+            dependency excludes - variables never included in any checksum.
+            This example uses variables from OpenEmbedded to help illustrate
+            the concept:
+            <literallayout class='monospaced'>
+     BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
+         SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
+         USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
+         PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
+         CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
+            </literallayout>
+            The previous example excludes the work directory, which is part of
+            <filename>TMPDIR</filename>.
+        </para>
+
+        <para>
+            The rules for deciding which hashes of dependent tasks to include through
+            dependency chains are more complex and are generally accomplished with a
+            Python function.
+            The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
+            of this and also illustrates how you can insert your own policy into the system
+            if so desired.
+            This file defines the two basic signature generators OpenEmbedded Core
+            uses:  "OEBasic" and "OEBasicHash".
+            By default, there is a dummy "noop" signature handler enabled in BitBake.
+            This means that behavior is unchanged from previous versions.
+            <filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default
+            through this setting in the <filename>bitbake.conf</filename> file:
+            <literallayout class='monospaced'>
+     BB_SIGNATURE_HANDLER ?= "OEBasicHash"
+            </literallayout>
+            The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
+            "OEBasic" version but adds the task hash to the stamp files.
+            This results in any metadata change that changes the task hash, automatically
+            causing the task to be run again.
+            This removes the need to bump
+            <link linkend='var-PR'><filename>PR</filename></link>
+            values, and changes to metadata automatically ripple across the build.
+        </para>
+
+        <para>
+            It is also worth noting that the end result of these signature generators is to
+            make some dependency and hash information available to the build.
+            This information includes:
+            <itemizedlist>
+                <listitem><para><filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
+                    The base hashes for each task in the recipe.
+                    </para></listitem>
+                <listitem><para><filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
+                    The base hashes for each dependent task.
+                    </para></listitem>
+                <listitem><para><filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
+                    The task dependencies for each task.
+                    </para></listitem>
+                <listitem><para><filename>BB_TASKHASH</filename>:
+                    The hash of the currently running task.
+                    </para></listitem>
+            </itemizedlist>
+        </para>
+
+        <para>
+            It is worth noting that BitBake's "-S" option lets you
+            debug Bitbake's processing of signatures.
+            The options passed to -S allow different debugging modes
+            to be used, either using BitBake's own debug functions
+            or possibly those defined in the metadata/signature handler
+            itself.
+            The simplest parameter to pass is "none", which causes a
+            set of signature information to be written out into
+            <filename>STAMP_DIR</filename>
+            corresponding to the targets specified.
+            The other currently available parameter is "printdiff",
+            which causes BitBake to try to establish the closest
+            signature match it can (e.g. in the sstate cache) and then
+            run <filename>bitbake-diffsigs</filename> over the matches
+            to determine the stamps and delta where these two
+            stamp trees diverge.
+            <note>
+                It is likely that future versions of BitBake will
+                provide other signature handlers triggered through
+                additional "-S" parameters.
+            </note>
+        </para>
+
+        <para>
+            You can find more information on checksum metadata in the
+            "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
+            section.
+        </para>
+    </section>
+
+    <section id='setscene'>
+        <title>Setscene</title>
+
+        <para>
+            The setscene process enables BitBake to handle "pre-built" artifacts.
+            The ability to handle and reuse these artifacts allows BitBake
+            the luxury of not having to build something from scratch every time.
+            Instead, BitBake can use, when possible, existing build artifacts.
+        </para>
+
+        <para>
+            BitBake needs to have reliable data indicating whether or not an
+            artifact is compatible.
+            Signatures, described in the previous section, provide an ideal
+            way of representing whether an artifact is compatible.
+            If a signature is the same, an object can be reused.
+        </para>
+
+        <para>
+            If an object can be reused, the problem then becomes how to
+            replace a given task or set of tasks with the pre-built artifact.
+            BitBake solves the problem with the "setscene" process.
+        </para>
+
+        <para>
+            When BitBake is asked to build a given target, before building anything,
+            it first asks whether cached information is available for any of the
+            targets it's building, or any of the intermediate targets.
+            If cached information is available, BitBake uses this information instead of
+            running the main tasks.
+        </para>
+
+        <para>
+            BitBake first calls the function defined by the
+            <link linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
+            variable with a list of tasks and corresponding
+            hashes it wants to build.
+            This function is designed to be fast and returns a list
+            of the tasks for which it believes in can obtain artifacts.
+        </para>
+
+        <para>
+            Next, for each of the tasks that were returned as possibilities,
+            BitBake executes a setscene version of the task that the possible
+            artifact covers.
+            Setscene versions of a task have the string "_setscene" appended to the
+            task name.
+            So, for example, the task with the name <filename>xxx</filename> has
+            a setscene task named <filename>xxx_setscene</filename>.
+            The setscene version of the task executes and provides the necessary
+            artifacts returning either success or failure.
+        </para>
+
+        <para>
+            As previously mentioned, an artifact can cover more than one task.
+            For example, it is pointless to obtain a compiler if you
+            already have the compiled binary.
+            To handle this, BitBake calls the
+            <link linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
+            function for each successful setscene task to know whether or not it needs
+            to obtain the dependencies of that task.
+        </para>
+
+        <para>
+            Finally, after all the setscene tasks have executed, BitBake calls the
+            function listed in
+            <link linkend='var-BB_SETSCENE_VERIFY_FUNCTION'><filename>BB_SETSCENE_VERIFY_FUNCTION</filename></link>
+            with the list of tasks BitBake thinks has been "covered".
+            The metadata can then ensure that this list is correct and can
+            inform BitBake that it wants specific tasks to be run regardless
+            of the setscene result.
+        </para>
+
+        <para>
+            You can find more information on setscene metadata in the
+            "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>"
+            section.
+        </para>
+    </section>
+</chapter>

bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml

@@ -0,0 +1,765 @@
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+
+<chapter>
+<title>File Download Support</title>
+
+    <para>
+        BitBake's fetch module is a standalone piece of library code
+        that deals with the intricacies of downloading source code
+        and files from remote systems.
+        Fetching source code is one of the cornerstones of building software.
+        As such, this module forms an important part of BitBake.
+    </para>
+
+    <para>
+        The current fetch module is called "fetch2" and refers to the
+        fact that it is the second major version of the API.
+        The original version is obsolete and has been removed from the codebase.
+        Thus, in all cases, "fetch" refers to "fetch2" in this
+        manual.
+    </para>
+
+    <section id='the-download-fetch'>
+        <title>The Download (Fetch)</title>
+
+        <para>
+            BitBake takes several steps when fetching source code or files.
+            The fetcher codebase deals with two distinct processes in order:
+            obtaining the files from somewhere (cached or otherwise)
+            and then unpacking those files into a specific location and
+            perhaps in a specific way.
+            Getting and unpacking the files is often optionally followed
+            by patching.
+            Patching, however, is not covered by this module.
+        </para>
+
+        <para>
+            The code to execute the first part of this process, a fetch,
+            looks something like the following:
+            <literallayout class='monospaced'>
+     src_uri = (d.getVar('SRC_URI', True) or "").split()
+     fetcher = bb.fetch2.Fetch(src_uri, d)
+     fetcher.download()
+            </literallayout>
+            This code sets up an instance of the fetch class.
+            The instance uses a space-separated list of URLs from the
+            <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
+            variable and then calls the <filename>download</filename>
+            method to download the files.
+        </para>
+
+        <para>
+            The instantiation of the fetch class is usually followed by:
+            <literallayout class='monospaced'>
+     rootdir = l.getVar('WORKDIR', True)
+     fetcher.unpack(rootdir)
+            </literallayout>
+            This code unpacks the downloaded files to the
+            specified by <filename>WORKDIR</filename>.
+            <note>
+                For convenience, the naming in these examples matches
+                the variables used by OpenEmbedded.
+                If you want to see the above code in action, examine
+                the OpenEmbedded class file <filename>base.bbclass</filename>.
+            </note>
+            The <filename>SRC_URI</filename> and <filename>WORKDIR</filename>
+            variables are not hardcoded into the fetcher, since those fetcher
+            methods can be (and are) called with different variable names.
+            In OpenEmbedded for example, the shared state (sstate) code uses
+            the fetch module to fetch the sstate files.
+        </para>
+
+        <para>
+            When the <filename>download()</filename> method is called,
+            BitBake tries to resolve the URLs by looking for source files
+            in a specific search order:
+            <itemizedlist>
+                <listitem><para><emphasis>Pre-mirror Sites:</emphasis>
+                    BitBake first uses pre-mirrors to try and find source files.
+                    These locations are defined using the
+                    <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
+                    variable.
+                    </para></listitem>
+                <listitem><para><emphasis>Source URI:</emphasis>
+                    If pre-mirrors fail, BitBake uses the original URL (e.g from
+                    <filename>SRC_URI</filename>).
+                    </para></listitem>
+                <listitem><para><emphasis>Mirror Sites:</emphasis>
+                    If fetch failures occur, BitBake next uses mirror locations as
+                    defined by the
+                    <link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
+                    variable.
+                    </para></listitem>
+            </itemizedlist>
+        </para>
+
+        <para>
+            For each URL passed to the fetcher, the fetcher
+            calls the submodule that handles that particular URL type.
+            This behavior can be the source of some confusion when you
+            are providing URLs for the <filename>SRC_URI</filename>
+            variable.
+            Consider the following two URLs:
+            <literallayout class='monospaced'>
+     http://git.yoctoproject.org/git/poky;protocol=git
+     git://git.yoctoproject.org/git/poky;protocol=http
+            </literallayout>
+            In the former case, the URL is passed to the
+            <filename>wget</filename> fetcher, which does not
+            understand "git".
+            Therefore, the latter case is the correct form since the
+            Git fetcher does know how to use HTTP as a transport.
+        </para>
+
+        <para>
+            Here are some examples that show commonly used mirror
+            definitions:
+            <literallayout class='monospaced'>
+     PREMIRRORS ?= "\
+         bzr://.*/.*   http://somemirror.org/sources/ \n \
+         cvs://.*/.*   http://somemirror.org/sources/ \n \
+         git://.*/.*   http://somemirror.org/sources/ \n \
+         hg://.*/.*    http://somemirror.org/sources/ \n \
+         osc://.*/.*   http://somemirror.org/sources/ \n \
+         p4://.*/.*    http://somemirror.org/sources/ \n \
+         svn://.*/.*   http://somemirror.org/sources/ \n"
+
+     MIRRORS =+ "\
+         ftp://.*/.*      http://somemirror.org/sources/ \n \
+         http://.*/.*     http://somemirror.org/sources/ \n \
+         https://.*/.*    http://somemirror.org/sources/ \n"
+            </literallayout>
+            It is useful to note that BitBake supports
+            cross-URLs.
+            It is possible to mirror a Git repository on an HTTP
+            server as a tarball.
+            This is what the <filename>git://</filename> mapping in
+            the previous example does.
+        </para>
+
+        <para>
+            Since network accesses are slow, Bitbake maintains a
+            cache of files downloaded from the network.
+            Any source files that are not local (i.e.
+            downloaded from the Internet) are placed into the download
+            directory, which is specified by the
+            <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
+            variable.
+        </para>
+
+        <para>
+            File integrity is of key importance for reproducing builds.
+            For non-local archive downloads, the fetcher code can verify
+            SHA-256 and MD5 checksums to ensure the archives have been
+            downloaded correctly.
+            You can specify these checksums by using the
+            <filename>SRC_URI</filename> variable with the appropriate
+            varflags as follows:
+            <literallayout class='monospaced'>
+     SRC_URI[md5sum] = "<replaceable>value</replaceable>"
+     SRC_URI[sha256sum] = "<replaceable>value</replaceable>"
+            </literallayout>
+            You can also specify the checksums as parameters on the
+            <filename>SRC_URI</filename> as shown below:
+            <literallayout class='monospaced'>
+     SRC_URI = "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d"
+            </literallayout>
+            If multiple URIs exist, you can specify the checksums either
+            directly as in the previous example, or you can name the URLs.
+            The following syntax shows how you name the URIs:
+            <literallayout class='monospaced'>
+     SRC_URI = "http://example.com/foobar.tar.bz2;name=foo"
+     SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d
+            </literallayout>
+            After a file has been downloaded and has had its checksum checked,
+            a ".done" stamp is placed in <filename>DL_DIR</filename>.
+            BitBake uses this stamp during subsequent builds to avoid
+            downloading or comparing a checksum for the file again.
+            <note>
+                It is assumed that local storage is safe from data corruption.
+                If this were not the case, there would be bigger issues to worry about.
+            </note>
+        </para>
+
+        <para>
+            If
+            <link linkend='var-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
+            is set, any download without a checksum triggers an
+            error message.
+            The
+            <link linkend='var-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
+            variable can be used to make any attempted network access a fatal
+            error, which is useful for checking that mirrors are complete
+            as well as other things.
+        </para>
+    </section>
+
+    <section id='bb-the-unpack'>
+        <title>The Unpack</title>
+
+        <para>
+            The unpack process usually immediately follows the download.
+            For all URLs except Git URLs, BitBake uses the common
+            <filename>unpack</filename> method.
+        </para>
+
+        <para>
+            A number of parameters exist that you can specify within the
+            URL to govern the behavior of the unpack stage:
+            <itemizedlist>
+                <listitem><para><emphasis>unpack:</emphasis>
+                    Controls whether the URL components are unpacked.
+                    If set to "1", which is the default, the components
+                    are unpacked.
+                    If set to "0", the unpack stage leaves the file alone.
+                    This parameter is useful when you want an archive to be
+                    copied in and not be unpacked.
+                    </para></listitem>
+                <listitem><para><emphasis>dos:</emphasis>
+                    Applies to <filename>.zip</filename> and
+                    <filename>.jar</filename> files and specifies whether to
+                    use DOS line ending conversion on text files.
+                    </para></listitem>
+                <listitem><para><emphasis>basepath:</emphasis>
+                    Instructs the unpack stage to strip the specified
+                    directories from the source path when unpacking.
+                    </para></listitem>
+                <listitem><para><emphasis>subdir:</emphasis>
+                    Unpacks the specific URL to the specified subdirectory
+                    within the root directory.
+                    </para></listitem>
+            </itemizedlist>
+            The unpack call automatically decompresses and extracts files
+            with ".Z", ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm".
+            ".srpm", ".deb" and ".bz2" extensions as well as various combinations
+            of tarball extensions.
+        </para>
+
+        <para>
+            As mentioned, the Git fetcher has its own unpack method that
+            is optimized to work with Git trees.
+            Basically, this method works by cloning the tree into the final
+            directory.
+            The process is completed using references so that there is
+            only one central copy of the Git metadata needed.
+        </para>
+    </section>
+
+    <section id='bb-fetchers'>
+        <title>Fetchers</title>
+
+        <para>
+            As mentioned earlier, the URL prefix determines which
+            fetcher submodule BitBake uses.
+            Each submodule can support different URL parameters,
+            which are described in the following sections.
+        </para>
+
+        <section id='local-file-fetcher'>
+            <title>Local file fetcher (<filename>file://</filename>)</title>
+
+            <para>
+                This submodule handles URLs that begin with
+                <filename>file://</filename>.
+                The filename you specify within the URL can be
+                either an absolute or relative path to a file.
+                If the filename is relative, the contents of the
+                <link linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
+                variable is used in the same way
+                <filename>PATH</filename> is used to find executables.
+                Failing that,
+                <link linkend='var-FILESDIR'><filename>FILESDIR</filename></link>
+                is used to find the appropriate relative file.
+                <note>
+                    <filename>FILESDIR</filename> is deprecated and can
+                    be replaced with <filename>FILESPATH</filename>.
+                    Because <filename>FILESDIR</filename> is likely to be
+                    removed, you should not use this variable in any new code.
+                </note>
+                If the file cannot be found, it is assumed that it is available in
+                <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
+                by the time the <filename>download()</filename> method is called.
+            </para>
+
+            <para>
+                If you specify a directory, the entire directory is
+                unpacked.
+            </para>
+
+            <para>
+                Here are a couple of example URLs, the first relative and
+                the second absolute:
+                <literallayout class='monospaced'>
+     SRC_URI = "file://relativefile.patch"
+     SRC_URI = "file:///Users/ich/very_important_software"
+                </literallayout>
+            </para>
+        </section>
+
+        <section id='http-ftp-fetcher'>
+            <title>HTTP/FTP wget fetcher (<filename>http://</filename>, <filename>ftp://</filename>, <filename>https://</filename>)</title>
+
+            <para>
+                This fetcher obtains files from web and FTP servers.
+                Internally, the fetcher uses the wget utility.
+            </para>
+
+            <para>
+                The executable and parameters used are specified by the
+                <filename>FETCHCMD_wget</filename> variable, which defaults
+                to sensible values.
+                The fetcher supports a parameter "downloadfilename" that
+                allows the name of the downloaded file to be specified.
+                Specifying the name of the downloaded file is useful
+                for avoiding collisions in
+                <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
+                when dealing with multiple files that have the same name.
+            </para>
+
+            <para>
+                Some example URLs are as follows:
+                <literallayout class='monospaced'>
+     SRC_URI = "http://oe.handhelds.org/not_there.aac"
+     SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac"
+     SRC_URI = "ftp://you@oe.handhelds.org/home/you/secret.plan"
+                </literallayout>
+            </para>
+            <note>
+               Because URL parameters are delimited by semi-colons, this can
+               introduce ambiguity when parsing URLs that also contain semi-colons,
+               for example:
+                <literallayout class='monospaced'>
+     SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"
+                </literallayout>
+               Such URLs should should be modified by replacing semi-colons with '&amp;' characters:
+               <literallayout class='monospaced'>
+     SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&amp;a=snapshot&amp;h=a5dd47"
+                </literallayout>
+                In most cases this should work. Treating semi-colons and '&amp;' in queries
+                identically is recommended by the World Wide Web Consortium (W3C).
+                Note that due to the nature of the URL, you may have to specify the name
+                of the downloaded file as well:
+              <literallayout class='monospaced'>
+     SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&amp;a=snapshot&amp;h=a5dd47;downloadfilename=myfile.bz2"
+                </literallayout>
+            </note>
+        </section>
+
+        <section id='cvs-fetcher'>
+            <title>CVS fetcher (<filename>(cvs://</filename>)</title>
+
+            <para>
+                This submodule handles checking out files from the
+                CVS version control system.
+                You can configure it using a number of different variables:
+                <itemizedlist>
+                    <listitem><para><emphasis><filename>FETCHCMD_cvs</filename>:</emphasis>
+                        The name of the executable to use when running
+                        the <filename>cvs</filename> command.
+                        This name is usually "cvs".
+                        </para></listitem>
+                    <listitem><para><emphasis><filename>SRCDATE</filename>:</emphasis>
+                        The date to use when fetching the CVS source code.
+                        A special value of "now" causes the checkout to
+                        be updated on every build.
+                        </para></listitem>
+                    <listitem><para><emphasis><link linkend='var-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
+                        Specifies where a temporary checkout is saved.
+                        The location is often <filename>DL_DIR/cvs</filename>.
+                        </para></listitem>
+                    <listitem><para><emphasis><filename>CVS_PROXY_HOST</filename>:</emphasis>
+                        The name to use as a "proxy=" parameter to the
+                        <filename>cvs</filename> command.
+                        </para></listitem>
+                    <listitem><para><emphasis><filename>CVS_PROXY_PORT</filename>:</emphasis>
+                        The port number to use as a "proxyport=" parameter to
+                        the <filename>cvs</filename> command.
+                        </para></listitem>
+                </itemizedlist>
+                As well as the standard username and password URL syntax,
+                you can also configure the fetcher with various URL parameters:
+            </para>
+
+            <para>
+                The supported parameters are as follows:
+                <itemizedlist>
+                    <listitem><para><emphasis>"method":</emphasis>
+                        The protocol over which to communicate with the CVS server.
+                        By default, this protocol is "pserver".
+                        If "method" is set to "ext", BitBake examines the
+                        "rsh" parameter and sets <filename>CVS_RSH</filename>.
+                        You can use "dir" for local directories.
+                        </para></listitem>
+                    <listitem><para><emphasis>"module":</emphasis>
+                        Specifies the module to check out.
+                        You must supply this parameter.
+                        </para></listitem>
+                    <listitem><para><emphasis>"tag":</emphasis>
+                        Describes which CVS TAG should be used for
+                        the checkout.
+                        By default, the TAG is empty.
+                        </para></listitem>
+                    <listitem><para><emphasis>"date":</emphasis>
+                        Specifies a date.
+                        If no "date" is specified, the
+                        <link linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
+                        of the configuration is used to checkout a specific date.
+                        The special value of "now" causes the checkout to be
+                        updated on every build.
+                        </para></listitem>
+                    <listitem><para><emphasis>"localdir":</emphasis>
+                        Used to rename the module.
+                        Effectively, you are renaming the output directory
+                        to which the module is unpacked.
+                        You are forcing the module into a special
+                        directory relative to
+                        <link linkend='var-CVSDIR'><filename>CVSDIR</filename></link>.
+                        </para></listitem>
+                    <listitem><para><emphasis>"rsh"</emphasis>
+                        Used in conjunction with the "method" parameter.
+                        </para></listitem>
+                    <listitem><para><emphasis>"scmdata":</emphasis>
+                        Causes the CVS metadata to be maintained in the tarball
+                        the fetcher creates when set to "keep".
+                        The tarball is expanded into the work directory.
+                        By default, the CVS metadata is removed.
+                        </para></listitem>
+                    <listitem><para><emphasis>"fullpath":</emphasis>
+                        Controls whether the resulting checkout is at the
+                        module level, which is the default, or is at deeper
+                        paths.
+                        </para></listitem>
+                    <listitem><para><emphasis>"norecurse":</emphasis>
+                        Causes the fetcher to only checkout the specified
+                        directory with no recurse into any subdirectories.
+                        </para></listitem>
+                    <listitem><para><emphasis>"port":</emphasis>
+                        The port to which the CVS server connects.
+                        </para></listitem>
+                </itemizedlist>
+                Some example URLs are as follows:
+                <literallayout class='monospaced'>
+     SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
+     SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
+                </literallayout>
+            </para>
+        </section>
+
+        <section id='svn-fetcher'>
+            <title>Subversion (SVN) Fetcher (<filename>svn://</filename>)</title>
+
+            <para>
+                This fetcher submodule fetches code from the
+                Subversion source control system.
+                The executable used is specified by
+                <filename>FETCHCMD_svn</filename>, which defaults
+                to "svn".
+                The fetcher's temporary working directory is set by
+                <link linkend='var-SVNDIR'><filename>SVNDIR</filename></link>,
+                which is usually <filename>DL_DIR/svn</filename>.
+            </para>
+
+            <para>
+                The supported parameters are as follows:
+                <itemizedlist>
+                    <listitem><para><emphasis>"module":</emphasis>
+                        The name of the svn module to checkout.
+                        You must provide this parameter.
+                        You can think of this parameter as the top-level
+                        directory of the repository data you want.
+                        </para></listitem>
+                    <listitem><para><emphasis>"protocol":</emphasis>
+                        The protocol to use, which defaults to "svn".
+                        Other options are "svn+ssh" and "rsh".
+                        For "rsh", the "rsh" parameter is also used.
+                        </para></listitem>
+                    <listitem><para><emphasis>"rev":</emphasis>
+                        The revision of the source code to checkout.
+                        </para></listitem>
+                    <listitem><para><emphasis>"date":</emphasis>
+                        The date of the source code to checkout.
+                        Specific revisions are generally much safer to checkout
+                        rather than by date as they do not involve timezones
+                        (e.g. they are much more deterministic).
+                        </para></listitem>
+                    <listitem><para><emphasis>"scmdata":</emphasis>
+                        Causes the “.svn” directories to be available during
+                        compile-time when set to "keep".
+                        By default, these directories are removed.
+                        </para></listitem>
+                    <listitem><para><emphasis>"transportuser":</emphasis>
+                        When required, sets the username for the transport.
+                        By default, this parameter is empty.
+                        The transport username is different than the username
+                        used in the main URL, which is passed to the subversion
+                        command.
+                        </para></listitem>
+                </itemizedlist>
+                Following are two examples using svn:
+                <literallayout class='monospaced'>
+     SRC_URI = "svn://svn.oe.handhelds.org/svn;module=vip;proto=http;rev=667"
+     SRC_URI = "svn://svn.oe.handhelds.org/svn/;module=opie;proto=svn+ssh;date=20060126"
+                </literallayout>
+            </para>
+        </section>
+
+        <section id='git-fetcher'>
+            <title>Git Fetcher (<filename>git://</filename>)</title>
+
+            <para>
+                This fetcher submodule fetches code from the Git
+                source control system.
+                The fetcher works by creating a bare clone of the
+                remote into
+                <link linkend='var-GITDIR'><filename>GITDIR</filename></link>,
+                which is usually <filename>DL_DIR/git2</filename>.
+                This bare clone is then cloned into the work directory during the
+                unpack stage when a specific tree is checked out.
+                This is done using alternates and by reference to
+                minimize the amount of duplicate data on the disk and
+                make the unpack process fast.
+                The executable used can be set with
+                <filename>FETCHCMD_git</filename>.
+            </para>
+
+            <para>
+                This fetcher supports the following parameters:
+                <itemizedlist>
+                    <listitem><para><emphasis>"protocol":</emphasis>
+                        The protocol used to fetch the files.
+                        The default is "git" when a hostname is set.
+                        If a hostname is not set, the Git protocol is "file".
+                        You can also use "http", "https", "ssh" and "rsync".
+                        </para></listitem>
+                    <listitem><para><emphasis>"nocheckout":</emphasis>
+                        Tells the fetcher to not checkout source code when
+                        unpacking when set to "1".
+                        Set this option for the URL where there is a custom
+                        routine to checkout code.
+                        The default is "0".
+                        </para></listitem>
+                    <listitem><para><emphasis>"rebaseable":</emphasis>
+                        Indicates that the upstream Git repository can be rebased.
+                        You should set this parameter to "1" if
+                        revisions can become detached from branches.
+                        In this case, the source mirror tarball is done per
+                        revision, which has a loss of efficiency.
+                        Rebasing the upstream Git repository could cause the
+                        current revision to disappear from the upstream repository.
+                        This option reminds the fetcher to preserve the local cache
+                        carefully for future use.
+                        The default value for this parameter is "0".
+                        </para></listitem>
+                    <listitem><para><emphasis>"nobranch":</emphasis>
+                        Tells the fetcher to not check the SHA validation
+                        for the branch when set to "1".
+                        The default is "0".
+                        Set this option for the recipe that refers to
+                        the commit that is valid for a tag instead of
+                        the branch.
+                        </para></listitem>
+                    <listitem><para><emphasis>"bareclone":</emphasis>
+                        Tells the fetcher to clone a bare clone into the
+                        destination directory without checking out a working tree.
+                        Only the raw Git metadata is provided.
+                        This parameter implies the "nocheckout" parameter as well.
+                        </para></listitem>
+                    <listitem><para><emphasis>"branch":</emphasis>
+                        The branch(es) of the Git tree to clone.
+                        If unset, this is assumed to be "master".
+                        The number of branch parameters much match the number of
+                        name parameters.
+                        </para></listitem>
+                    <listitem><para><emphasis>"rev":</emphasis>
+                        The revision to use for the checkout.
+                        The default is "master".
+                        </para></listitem>
+                    <listitem><para><emphasis>"tag":</emphasis>
+                        Specifies a tag to use for the checkout.
+                        To correctly resolve tags, BitBake must access the
+                        network.
+                        For that reason, tags are often not used.
+                        As far as Git is concerned, the "tag" parameter behaves
+                        effectively the same as the "rev" parameter.
+                        </para></listitem>
+                    <listitem><para><emphasis>"subpath":</emphasis>
+                        Limits the checkout to a specific subpath of the tree.
+                        By default, the whole tree is checked out.
+                        </para></listitem>
+                    <listitem><para><emphasis>"destsuffix":</emphasis>
+                        The name of the path in which to place the checkout.
+                        By default, the path is <filename>git/</filename>.
+                        </para></listitem>
+                </itemizedlist>
+                Here are some example URLs:
+                <literallayout class='monospaced'>
+     SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
+     SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http"
+                </literallayout>
+            </para>
+        </section>
+
+        <section id='gitsm-fetcher'>
+            <title>Git Submodule Fetcher (<filename>gitsm://</filename>)</title>
+
+            <para>
+                This fetcher submodule inherits from the
+                <link linkend='git-fetcher'>Git fetcher</link> and extends
+                that fetcher's behavior by fetching a repository's submodules.
+                <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
+                is passed to the Git fetcher as described in the
+                "<link linkend='git-fetcher'>Git Fetcher (<filename>git://</filename>)</link>"
+                section.
+                <note>
+                    <title>Notes and Warnings</title>
+                    <para>
+                        You must clean a recipe when switching between
+                        '<filename>git://</filename>' and
+                        '<filename>gitsm://</filename>' URLs.
+                    </para>
+
+                    <para>
+                        The Git Submodules fetcher is not a complete fetcher
+                        implementation.
+                        The fetcher has known issues where it does not use the
+                        normal source mirroring infrastructure properly.
+                    </para>
+                </note>
+            </para>
+        </section>
+
+        <section id='clearcase-fetcher'>
+            <title>ClearCase Fetcher (<filename>ccrc://</filename>)</title>
+
+            <para>
+                This fetcher submodule fetches code from a
+                <ulink url='http://en.wikipedia.org/wiki/Rational_ClearCase'>ClearCase</ulink>
+                repository.
+            </para>
+
+            <para>
+                To use this fetcher, make sure your recipe has proper
+                <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
+                <link linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
+                <link linkend='var-PV'><filename>PV</filename></link> settings.
+                Here is an example:
+                <literallayout class='monospaced'>
+     SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
+     SRCREV = "EXAMPLE_CLEARCASE_TAG"
+     PV = "${@d.getVar("SRCREV", False).replace("/", "+")}"
+                </literallayout>
+                The fetcher uses the <filename>rcleartool</filename> or
+                <filename>cleartool</filename> remote client, depending on
+                which one is available.
+            </para>
+
+            <para>
+                Following are options for the <filename>SRC_URI</filename>
+                statement:
+                <itemizedlist>
+                    <listitem><para><emphasis><filename>vob</filename></emphasis>:
+                        The name, which must include the
+                        prepending "/" character, of the ClearCase VOB.
+                        This option is required.
+                        </para></listitem>
+                    <listitem><para><emphasis><filename>module</filename></emphasis>:
+                        The module, which must include the
+                        prepending "/" character, in the selected VOB.
+                        <note>
+                            The <filename>module</filename> and <filename>vob</filename>
+                            options are combined to create the <filename>load</filename> rule in
+                            the view config spec.
+                            As an example, consider the <filename>vob</filename> and 
+                            <filename>module</filename> values from the 
+                            <filename>SRC_URI</filename> statement at the start of this section.
+                            Combining those values results in the following:
+                            <literallayout class='monospaced'>
+     load /example_vob/example_module
+                            </literallayout>
+                        </note>
+                        </para></listitem>
+                    <listitem><para><emphasis><filename>proto</filename></emphasis>:
+                        The protocol, which can be either <filename>http</filename> or
+                        <filename>https</filename>.
+                        </para></listitem>
+                </itemizedlist>
+            </para>
+
+            <para>
+                By default, the fetcher creates a configuration specification.
+                If you want this specification written to an area other than the default,
+                use the <filename>CCASE_CUSTOM_CONFIG_SPEC</filename> variable
+                in your recipe to define where the specification is written.
+                <note>
+                    the <filename>SRCREV</filename> loses its functionality if you
+                    specify this variable.
+                    However, <filename>SRCREV</filename> is still used to label the
+                    archive after a fetch even though it does not define what is
+                    fetched.
+                </note>
+            </para>
+
+            <para>
+                Here are a couple of other behaviors worth mentioning:
+                <itemizedlist>
+                    <listitem><para>
+                        When using <filename>cleartool</filename>, the login of
+                        <filename>cleartool</filename> is handled by the system.
+                        The login require no special steps.
+                        </para></listitem>
+                    <listitem><para>
+                        In order to use <filename>rcleartool</filename> with authenticated
+                        users, an "rcleartool login" is necessary before using the fetcher.
+                        </para></listitem>
+                </itemizedlist>
+            </para>
+        </section>
+
+        <section id='other-fetchers'>
+            <title>Other Fetchers</title>
+
+            <para>
+                Fetch submodules also exist for the following:
+                <itemizedlist>
+                    <listitem><para>
+                        Bazaar (<filename>bzr://</filename>)
+                        </para></listitem>
+                    <listitem><para>
+                        Perforce (<filename>p4://</filename>)
+                        </para></listitem>
+                    <listitem><para>
+                        Trees using Git Annex (<filename>gitannex://</filename>)
+                        </para></listitem>
+                    <listitem><para>
+                        Secure FTP (<filename>sftp://</filename>)
+                        </para></listitem>
+                    <listitem><para>
+                        Secure Shell (<filename>ssh://</filename>)
+                        </para></listitem>
+                    <listitem><para>
+                        Repo (<filename>repo://</filename>)
+                        </para></listitem>
+                    <listitem><para>
+                        OSC (<filename>osc://</filename>)
+                        </para></listitem>
+                    <listitem><para>
+                        Mercurial (<filename>hg://</filename>)
+                        </para></listitem>
+                </itemizedlist>
+                No documentation currently exists for these lesser used
+                fetcher submodules.
+                However, you might find the code helpful and readable.
+            </para>
+        </section>
+    </section>
+
+    <section id='auto-revisions'>
+        <title>Auto Revisions</title>
+
+        <para>
+            We need to document <filename>AUTOREV</filename> and
+            <filename>SRCREV_FORMAT</filename> here.
+        </para>
+    </section>
+</chapter>

bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml

@@ -0,0 +1,505 @@
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+
+<appendix id='hello-world-example'>
+    <title>Hello World Example</title>
+
+    <section id='bitbake-hello-world'>
+        <title>BitBake Hello World</title>
+
+        <para>
+            The simplest example commonly used to demonstrate any new
+            programming language or tool is the
+            "<ulink url="http://en.wikipedia.org/wiki/Hello_world_program">Hello World</ulink>"
+            example.
+            This appendix demonstrates, in tutorial form, Hello
+            World within the context of BitBake.
+            The tutorial describes how to create a new project
+            and the applicable metadata files necessary to allow
+            BitBake to build it.
+        </para>
+    </section>
+
+    <section id='example-obtaining-bitbake'>
+        <title>Obtaining BitBake</title>
+
+        <para>
+            See the
+            "<link linkend='obtaining-bitbake'>Obtaining BitBake</link>"
+            section for information on how to obtain BitBake.
+            Once you have the source code on your machine, the BitBake directory
+            appears as follows:
+            <literallayout class='monospaced'>
+     $ ls -al
+     total 100
+     drwxrwxr-x. 9 wmat wmat  4096 Jan 31 13:44 .
+     drwxrwxr-x. 3 wmat wmat  4096 Feb  4 10:45 ..
+     -rw-rw-r--. 1 wmat wmat   365 Nov 26 04:55 AUTHORS
+     drwxrwxr-x. 2 wmat wmat  4096 Nov 26 04:55 bin
+     drwxrwxr-x. 4 wmat wmat  4096 Jan 31 13:44 build
+     -rw-rw-r--. 1 wmat wmat 16501 Nov 26 04:55 ChangeLog
+     drwxrwxr-x. 2 wmat wmat  4096 Nov 26 04:55 classes
+     drwxrwxr-x. 2 wmat wmat  4096 Nov 26 04:55 conf
+     drwxrwxr-x. 3 wmat wmat  4096 Nov 26 04:55 contrib
+     -rw-rw-r--. 1 wmat wmat 17987 Nov 26 04:55 COPYING
+     drwxrwxr-x. 3 wmat wmat  4096 Nov 26 04:55 doc
+     -rw-rw-r--. 1 wmat wmat    69 Nov 26 04:55 .gitignore
+     -rw-rw-r--. 1 wmat wmat   849 Nov 26 04:55 HEADER
+     drwxrwxr-x. 5 wmat wmat  4096 Jan 31 13:44 lib
+     -rw-rw-r--. 1 wmat wmat   195 Nov 26 04:55 MANIFEST.in
+     -rw-rw-r--. 1 wmat wmat  2887 Nov 26 04:55 TODO
+            </literallayout>
+        </para>
+
+        <para>
+            At this point, you should have BitBake cloned to
+            a directory that matches the previous listing except for
+            dates and user names.
+        </para>
+    </section>
+
+    <section id='setting-up-the-bitbake-environment'>
+        <title>Setting Up the BitBake Environment</title>
+
+        <para>
+            First, you need to be sure that you can run BitBake.
+            Set your working directory to where your local BitBake
+            files are and run the following command:
+            <literallayout class='monospaced'>
+     $ ./bin/bitbake --version
+     BitBake Build Tool Core version 1.23.0, bitbake version 1.23.0
+            </literallayout>
+            The console output tells you what version you are running.
+        </para>
+
+        <para>
+            The recommended method to run BitBake is from a directory of your
+            choice.
+            To be able to run BitBake from any directory, you need to add the
+            executable binary to your binary to your shell's environment
+            <filename>PATH</filename> variable.
+            First, look at your current <filename>PATH</filename> variable
+            by entering the following:
+            <literallayout class='monospaced'>
+     $ echo $PATH
+            </literallayout>
+            Next, add the directory location for the BitBake binary to the
+            <filename>PATH</filename>.
+            Here is an example that adds the
+            <filename>/home/scott-lenovo/bitbake/bin</filename> directory
+            to the front of the <filename>PATH</filename> variable:
+            <literallayout class='monospaced'>
+     $ export PATH=/home/scott-lenovo/bitbake/bin:$PATH
+            </literallayout>
+            You should now be able to enter the <filename>bitbake</filename>
+            command from the command line while working from any directory.
+        </para>
+    </section>
+
+    <section id='the-hello-world-example'>
+        <title>The Hello World Example</title>
+
+        <para>
+            The overall goal of this exercise is to build a
+            complete "Hello World" example utilizing task and layer
+            concepts.
+            Because this is how modern projects such as OpenEmbedded and
+            the Yocto Project utilize BitBake, the example
+            provides an excellent starting point for understanding
+            BitBake.
+        </para>
+
+        <para>
+            To help you understand how to use BitBake to build targets,
+            the example starts with nothing but the <filename>bitbake</filename>
+            command, which causes BitBake to fail and report problems.
+            The example progresses by adding pieces to the build to
+            eventually conclude with a working, minimal "Hello World"
+            example.
+        </para>
+
+        <para>
+            While every attempt is made to explain what is happening during
+            the example, the descriptions cannot cover everything.
+            You can find further information throughout this manual.
+            Also, you can actively participate in the
+            <ulink url='http://lists.openembedded.org/mailman/listinfo/bitbake-devel'></ulink>
+            discussion mailing list about the BitBake build tool.
+        </para>
+
+        <note>
+            This example was inspired by and drew heavily from these sources:
+            <itemizedlist>
+                <listitem><para>
+                    <ulink url="http://www.mail-archive.com/yocto@yoctoproject.org/msg09379.html">Mailing List post - The BitBake equivalent of "Hello, World!"</ulink>
+                    </para></listitem>
+                <listitem><para>
+                    <ulink url="http://hambedded.org/blog/2012/11/24/from-bitbake-hello-world-to-an-image/">Hambedded Linux blog post - From Bitbake Hello World to an Image</ulink>
+                    </para></listitem>
+            </itemizedlist>
+        </note>
+
+        <para>
+            As stated earlier, the goal of this example
+            is to eventually compile "Hello World".
+            However, it is unknown what BitBake needs and what you have
+            to provide in order to achieve that goal.
+            Recall that BitBake utilizes three types of metadata files:
+            <link linkend='configuration-files'>Configuration Files</link>,
+            <link linkend='classes'>Classes</link>, and
+            <link linkend='recipes'>Recipes</link>.
+            But where do they go?
+            How does BitBake find them?
+            BitBake's error messaging helps you answer these types of questions
+            and helps you better understand exactly what is going on.
+        </para>
+
+        <para>
+            Following is the complete "Hello World" example.
+        </para>
+
+        <orderedlist>
+            <listitem><para><emphasis>Create a Project Directory:</emphasis>
+                First, set up a directory for the "Hello World" project.
+                Here is how you can do so in your home directory:
+                <literallayout class='monospaced'>
+     $ mkdir ~/hello
+     $ cd ~/hello
+                </literallayout>
+                This is the directory that BitBake will use to do all of
+                its work.
+                You can use this directory to keep all the metafiles needed
+                by BitBake.
+                Having a project directory is a good way to isolate your
+                project.
+                </para></listitem>
+            <listitem><para><emphasis>Run Bitbake:</emphasis>
+                At this point, you have nothing but a project directory.
+                Run the <filename>bitbake</filename> command and see what
+                it does:
+                <literallayout class='monospaced'>
+     $ bitbake
+     The BBPATH variable is not set and bitbake did not
+     find a conf/bblayers.conf file in the expected location.
+     Maybe you accidentally invoked bitbake from the wrong directory?
+     DEBUG: Removed the following variables from the environment:
+     GNOME_DESKTOP_SESSION_ID, XDG_CURRENT_DESKTOP,
+     GNOME_KEYRING_CONTROL, DISPLAY, SSH_AGENT_PID, LANG, no_proxy,
+     XDG_SESSION_PATH, XAUTHORITY, SESSION_MANAGER, SHLVL,
+     MANDATORY_PATH, COMPIZ_CONFIG_PROFILE, WINDOWID, EDITOR,
+     GPG_AGENT_INFO, SSH_AUTH_SOCK, GDMSESSION, GNOME_KEYRING_PID,
+     XDG_SEAT_PATH, XDG_CONFIG_DIRS, LESSOPEN, DBUS_SESSION_BUS_ADDRESS,
+     _, XDG_SESSION_COOKIE, DESKTOP_SESSION, LESSCLOSE, DEFAULTS_PATH,
+     UBUNTU_MENUPROXY, OLDPWD, XDG_DATA_DIRS, COLORTERM, LS_COLORS
+                </literallayout>
+                The majority of this output is specific to environment variables
+                that are not directly relevant to BitBake.
+                However, the very first message regarding the
+                <filename>BBPATH</filename> variable and the
+                <filename>conf/bblayers.conf</filename> file
+                is relevant.</para>
+                <para>
+                When you run BitBake, it begins looking for metadata files.
+                The
+                <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
+                variable is what tells BitBake where to look for those files.
+                <filename>BBPATH</filename> is not set and you need to set it.
+                Without <filename>BBPATH</filename>, Bitbake cannot
+                find any configuration files (<filename>.conf</filename>)
+                or recipe files (<filename>.bb</filename>) at all.
+                BitBake also cannot find the <filename>bitbake.conf</filename>
+                file.
+                </para></listitem>
+            <listitem><para><emphasis>Setting <filename>BBPATH</filename>:</emphasis>
+                For this example, you can set <filename>BBPATH</filename>
+                in the same manner that you set <filename>PATH</filename>
+                earlier in the appendix.
+                You should realize, though, that it is much more flexible to set the
+                <filename>BBPATH</filename> variable up in a configuration
+                file for each project.</para>
+                <para>From your shell, enter the following commands to set and
+                export the <filename>BBPATH</filename> variable:
+                <literallayout class='monospaced'>
+     $ BBPATH="<replaceable>projectdirectory</replaceable>"
+     $ export BBPATH
+                </literallayout>
+                Use your actual project directory in the command.
+                BitBake uses that directory to find the metadata it needs for
+                your project.
+                <note>
+                    When specifying your project directory, do not use the
+                    tilde ("~") character as BitBake does not expand that character
+                    as the shell would.
+                </note>
+                </para></listitem>
+            <listitem><para><emphasis>Run Bitbake:</emphasis>
+                Now that you have <filename>BBPATH</filename> defined, run
+                the <filename>bitbake</filename> command again:
+                <literallayout class='monospaced'>
+     $ bitbake
+     ERROR: Traceback (most recent call last):
+       File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 163, in wrapped
+         return func(fn, *args)
+       File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 173, in parse_config_file
+         return bb.parse.handle(fn, data, include)
+       File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line 99, in handle
+         return h['handle'](fn, data, include)
+       File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 120, in handle
+         abs_fn = resolve_file(fn, data)
+       File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line 117, in resolve_file
+         raise IOError("file %s not found in %s" % (fn, bbpath))
+     IOError: file conf/bitbake.conf not found in /home/scott-lenovo/hello
+
+     ERROR: Unable to parse conf/bitbake.conf: file conf/bitbake.conf not found in /home/scott-lenovo/hello
+                </literallayout>
+                This sample output shows that BitBake could not find the
+                <filename>conf/bitbake.conf</filename> file in the project
+                directory.
+                This file is the first thing BitBake must find in order
+                to build a target.
+                And, since the project directory for this example is
+                empty, you need to provide a <filename>conf/bitbake.conf</filename>
+                file.
+                </para></listitem>
+            <listitem><para><emphasis>Creating <filename>conf/bitbake.conf</filename>:</emphasis>
+                The <filename>conf/bitbake.conf</filename> includes a number of
+                configuration variables BitBake uses for metadata and recipe
+                files.
+                For this example, you need to create the file in your project directory
+                and define some key BitBake variables.
+                For more information on the <filename>bitbake.conf</filename>,
+                see
+                <ulink url='http://hambedded.org/blog/2012/11/24/from-bitbake-hello-world-to-an-image/#an-overview-of-bitbakeconf'></ulink>
+                </para>
+                <para>Use the following commands to create the <filename>conf</filename>
+                directory in the project directory:
+                <literallayout class='monospaced'>
+     $ mkdir conf
+                </literallayout>
+                From within the <filename>conf</filename> directory, use
+                some editor to create the <filename>bitbake.conf</filename>
+                so that it contains the following:
+                <literallayout class='monospaced'>
+     TMPDIR  = "${<link linkend='var-TOPDIR'>TOPDIR</link>}/tmp"
+     <link linkend='var-CACHE'>CACHE</link>   = "${TMPDIR}/cache"
+     <link linkend='var-STAMP'>STAMP</link>   = "${TMPDIR}/stamps"
+     <link linkend='var-T'>T</link>       = "${TMPDIR}/work"
+     <link linkend='var-B'>B</link>       = "${TMPDIR}"
+                </literallayout>
+                The <filename>TMPDIR</filename> variable establishes a directory
+                that BitBake uses for build output and intermediate files (other
+                than the cached information used by the
+                <link linkend='setscene'>Setscene</link> process.
+                Here, the <filename>TMPDIR</filename> directory is set to
+                <filename>hello/tmp</filename>.
+                <note><title>Tip</title>
+                    You can always safely delete the <filename>tmp</filename>
+                    directory in order to rebuild a BitBake target.
+                    The build process creates the directory for you
+                    when you run BitBake.
+                </note></para>
+                <para>For information about each of the other variables defined in this
+                example, click on the links to take you to the definitions in
+                the glossary.
+                </para></listitem>
+            <listitem><para><emphasis>Run Bitbake:</emphasis>
+                After making sure that the <filename>conf/bitbake.conf</filename>
+                file exists, you can run the <filename>bitbake</filename>
+                command again:
+                <literallayout class='monospaced'>
+$ bitbake
+ERROR: Traceback (most recent call last):
+  File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 163, in wrapped
+    return func(fn, *args)
+  File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 177, in _inherit
+    bb.parse.BBHandler.inherit(bbclass, "configuration INHERITs", 0, data)
+  File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 92, in inherit
+    include(fn, file, lineno, d, "inherit")
+  File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 100, in include
+    raise ParseError("Could not %(error_out)s file %(fn)s" % vars(), oldfn, lineno)
+ParseError: ParseError in configuration INHERITs: Could not inherit file classes/base.bbclass
+
+ERROR: Unable to parse base: ParseError in configuration INHERITs: Could not inherit file classes/base.bbclass
+                </literallayout>
+                In the sample output, BitBake could not find the
+                <filename>classes/base.bbclass</filename> file.
+                You need to create that file next.
+                </para></listitem>
+            <listitem><para><emphasis>Creating <filename>classes/base.bbclass</filename>:</emphasis>
+                BitBake uses class files to provide common code and functionality.
+                The minimally required class for BitBake is the
+                <filename>classes/base.bbclass</filename> file.
+                The <filename>base</filename> class is implicitly inherited by
+                every recipe.
+                BitBake looks for the class in the <filename>classes</filename>
+                directory of the project (i.e <filename>hello/classes</filename>
+                in this example).
+                </para>
+                <para>Create the <filename>classes</filename> directory as follows:
+                <literallayout class='monospaced'>
+     $ cd $HOME/hello
+     $ mkdir classes
+                </literallayout>
+                Move to the <filename>classes</filename> directory and then
+                create the <filename>base.bbclass</filename> file by inserting
+                this single line:
+                <literallayout class='monospaced'>
+     addtask build
+                </literallayout>
+                The minimal task that BitBake runs is the
+                <filename>do_build</filename> task.
+                This is all the example needs in order to build the project.
+                Of course, the <filename>base.bbclass</filename> can have much
+                more depending on which build environments BitBake is
+                supporting.
+                For more information on the <filename>base.bbclass</filename> file,
+                you can look at
+                <ulink url='http://hambedded.org/blog/2012/11/24/from-bitbake-hello-world-to-an-image/#tasks'></ulink>.
+                </para></listitem>
+            <listitem><para><emphasis>Run Bitbake:</emphasis>
+                After making sure that the <filename>classes/base.bbclass</filename>
+                file exists, you can run the <filename>bitbake</filename>
+                command again:
+                <literallayout class='monospaced'>
+     $ bitbake
+     Nothing to do.  Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.
+                </literallayout>
+                BitBake is finally reporting no errors.
+                However, you can see that it really does not have anything
+                to do.
+                You need to create a recipe that gives BitBake something to do.
+                </para></listitem>
+            <listitem><para><emphasis>Creating a Layer:</emphasis>
+                While it is not really necessary for such a small example,
+                it is good practice to create a layer in which to keep your
+                code separate from the general metadata used by BitBake.
+                Thus, this example creates and uses a layer called "mylayer".
+                <note>
+                    You can find additional information on adding a layer at
+                    <ulink url='http://hambedded.org/blog/2012/11/24/from-bitbake-hello-world-to-an-image/#adding-an-example-layer'></ulink>.
+                </note>
+                </para>
+                <para>Minimally, you need a recipe file and a layer configuration
+                file in your layer.
+                The configuration file needs to be in the <filename>conf</filename>
+                directory inside the layer.
+                Use these commands to set up the layer and the <filename>conf</filename>
+                directory:
+                <literallayout class='monospaced'>
+     $ cd $HOME
+     $ mkdir mylayer
+     $ cd mylayer
+     $ mkdir conf
+                </literallayout>
+                Move to the <filename>conf</filename> directory and create a
+                <filename>layer.conf</filename> file that has the following:
+                <literallayout class='monospaced'>
+     BBPATH .= ":${<link linkend='var-LAYERDIR'>LAYERDIR</link>}"
+
+     <link linkend='var-BBFILES'>BBFILES</link> += "${LAYERDIR}/*.bb"
+
+     <link linkend='var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</link> += "mylayer"
+     <link linkend='var-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer</link> := "^${LAYERDIR}/"
+                </literallayout>
+                For information on these variables, click the links
+                to go to the definitions in the glossary.</para>
+                <para>You need to create the recipe file next.
+                Inside your layer at the top-level, use an editor and create
+                a recipe file named <filename>printhello.bb</filename> that
+                has the following:
+                <literallayout class='monospaced'>
+     <link linkend='var-DESCRIPTION'>DESCRIPTION</link> = "Prints Hello World"
+     <link linkend='var-PN'>PN</link> = 'printhello'
+     <link linkend='var-PV'>PV</link> = '1'
+
+     python do_build() {
+        bb.plain("********************");
+        bb.plain("*                  *");
+        bb.plain("*  Hello, World!   *");
+        bb.plain("*                  *");
+        bb.plain("********************");
+     }
+                </literallayout>
+                The recipe file simply provides a description of the
+                recipe, the name, version, and the <filename>do_build</filename>
+                task, which prints out "Hello World" to the console.
+                For more information on these variables, follow the links
+                to the glossary.
+                </para></listitem>
+            <listitem><para><emphasis>Run Bitbake With a Target:</emphasis>
+                Now that a BitBake target exists, run the command and provide
+                that target:
+                <literallayout class='monospaced'>
+     $ cd $HOME/hello
+     $ bitbake printhello
+     ERROR: no recipe files to build, check your BBPATH and BBFILES?
+
+     Summary: There was 1 ERROR message shown, returning a non-zero exit code.
+                </literallayout>
+                We have created the layer with the recipe and the layer
+                configuration file but it still seems that BitBake cannot
+                find the recipe.
+                BitBake needs a <filename>conf/bblayers.conf</filename> that
+                lists the layers for the project.
+                Without this file, BitBake cannot find the recipe.
+                </para></listitem>
+            <listitem><para><emphasis>Creating <filename>conf/bblayers.conf</filename>:</emphasis>
+                BitBake uses the <filename>conf/bblayers.conf</filename> file
+                to locate layers needed for the project.
+                This file must reside in the <filename>conf</filename> directory
+                of the project (i.e. <filename>hello/conf</filename> for this
+                example).</para>
+                <para>Set your working directory to the <filename>hello/conf</filename>
+                directory and then create the <filename>bblayers.conf</filename>
+                file so that it contains the following:
+                <literallayout class='monospaced'>
+     BBLAYERS ?= " \
+       /home/&lt;you&gt;/mylayer \
+       "
+                </literallayout>
+                You need to provide your own information for
+                <filename>you</filename> in the file.
+                </para></listitem>
+            <listitem><para><emphasis>Run Bitbake With a Target:</emphasis>
+                Now that you have supplied the <filename>bblayers.conf</filename>
+                file, run the <filename>bitbake</filename> command and provide
+                the target:
+                <literallayout class='monospaced'>
+     $ bitbake printhello
+     Parsing recipes: 100% |##################################################################################|
+     Time: 00:00:00
+     Parsing of 1 .bb files complete (0 cached, 1 parsed). 1 targets, 0 skipped, 0 masked, 0 errors.
+     NOTE: Resolving any missing task queue dependencies
+     NOTE: Preparing RunQueue
+     NOTE: Executing RunQueue Tasks
+     ********************
+     *                  *
+     *  Hello, World!   *
+     *                  *
+     ********************
+     NOTE: Tasks Summary: Attempted 1 tasks of which 0 didn't need to be rerun and all succeeded.
+                </literallayout>
+                BitBake finds the <filename>printhello</filename> recipe and
+                successfully runs the task.
+                <note>
+                    After the first execution, re-running
+                    <filename>bitbake printhello</filename> again will not
+                    result in a BitBake run that prints the same console
+                    output.
+                    The reason for this is that the first time the
+                    <filename>printhello.bb</filename> recipe's
+                    <filename>do_build</filename> task executes
+                    successfully, BitBake writes a stamp file for the task.
+                    Thus, the next time you attempt to run the task
+                    using that same <filename>bitbake</filename> command,
+                    BitBake notices the stamp and therefore determines
+                    that the task does not need to be re-run.
+                    If you delete the <filename>tmp</filename> directory
+                    or run <filename>bitbake -c clean printhello</filename>
+                    and then re-run the build, the "Hello, World!" message will
+                    be printed again.
+                </note>
+                </para></listitem>
+        </orderedlist>
+    </section>
+</appendix>

bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml

@@ -0,0 +1,691 @@
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+    "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+
+<chapter id="bitbake-user-manual-intro">
+    <title>Overview</title>
+
+    <para>
+        Welcome to the BitBake User Manual.
+        This manual provides information on the BitBake tool.
+        The information attempts to be as independent as possible regarding
+        systems that use BitBake, such as OpenEmbedded and the
+        Yocto Project.
+        In some cases, scenarios or examples within the context of
+        a build system are used in the manual to help with understanding.
+        For these cases, the manual clearly states the context.
+    </para>
+
+    <section id="intro">
+        <title>Introduction</title>
+
+        <para>
+            Fundamentally, BitBake is a generic task execution
+            engine that allows shell and Python tasks to be run
+            efficiently and in parallel while working within
+            complex inter-task dependency constraints.
+            One of BitBake's main users, OpenEmbedded, takes this core
+            and builds embedded Linux software stacks using
+            a task-oriented approach.
+        </para>
+
+        <para>
+            Conceptually, BitBake is similar to GNU Make in
+            some regards but has significant differences:
+            <itemizedlist>
+                <listitem><para>
+                    BitBake executes tasks according to provided
+                    metadata that builds up the tasks.
+                    Metadata is stored in recipe (<filename>.bb</filename>)
+                    and related recipe "append" (<filename>.bbappend</filename>)
+                    files, configuration (<filename>.conf</filename>) and
+                    underlying include (<filename>.inc</filename>) files, and
+                    in class (<filename>.bbclass</filename>) files.
+                    The metadata provides
+                    BitBake with instructions on what tasks to run and
+                    the dependencies between those tasks.
+                    </para></listitem>
+                <listitem><para>
+                    BitBake includes a fetcher library for obtaining source
+                    code from various places such as local files, source control
+                    systems, or websites.
+                    </para></listitem>
+                <listitem><para>
+                    The instructions for each unit to be built (e.g. a piece
+                    of software) are known as "recipe" files and
+                    contain all the information about the unit
+                    (dependencies, source file locations, checksums, description
+                    and so on).
+                    </para></listitem>
+                <listitem><para>
+                    BitBake includes a client/server abstraction and can
+                    be used from a command line or used as a service over
+                    XML-RPC and has several different user interfaces.
+                    </para></listitem>
+            </itemizedlist>
+        </para>
+    </section>
+
+    <section id="history-and-goals">
+        <title>History and Goals</title>
+
+        <para>
+            BitBake was originally a part of the OpenEmbedded project.
+            It was inspired by the Portage package management system
+            used by the Gentoo Linux distribution.
+            On December 7, 2004, OpenEmbedded project team member
+            Chris Larson split the project into two distinct pieces:
+            <itemizedlist>
+                <listitem><para>BitBake, a generic task executor</para></listitem>
+                <listitem><para>OpenEmbedded, a metadata set utilized by
+                    BitBake</para></listitem>
+            </itemizedlist>
+            Today, BitBake is the primary basis of the
+            <ulink url="http://www.openembedded.org/">OpenEmbedded</ulink>
+            project, which is being used to build and maintain Linux
+            distributions such as the
+            <ulink url='http://www.angstrom-distribution.org/'>Angstrom Distribution</ulink>,
+            and which is also being used as the build tool for Linux projects
+            such as the
+            <ulink url='http://www.yoctoproject.org'>Yocto Project</ulink>.
+        </para>
+
+        <para>
+            Prior to BitBake, no other build tool adequately met the needs of
+            an aspiring embedded Linux distribution.
+            All of the build systems used by traditional desktop Linux
+            distributions lacked important functionality, and none of the
+            ad hoc Buildroot-based systems, prevalent in the
+            embedded space, were scalable or maintainable.
+        </para>
+
+        <para>
+            Some important original goals for BitBake were:
+            <itemizedlist>
+                <listitem><para>
+                    Handle cross-compilation.
+                    </para></listitem>
+                <listitem><para>
+                    Handle inter-package dependencies (build time on
+                    target architecture, build time on native
+                    architecture, and runtime).
+                    </para></listitem>
+                <listitem><para>
+                    Support running any number of tasks within a given
+                    package, including, but not limited to, fetching
+                    upstream sources, unpacking them, patching them,
+                    configuring them, and so forth.
+                    </para></listitem>
+                <listitem><para>
+                    Be Linux distribution agnostic for both build and
+                    target systems.
+                    </para></listitem>
+                <listitem><para>
+                    Be architecture agnostic.
+                    </para></listitem>
+                <listitem><para>
+                    Support multiple build and target operating systems
+                    (e.g. Cygwin, the BSDs, and so forth).
+                    </para></listitem>
+                <listitem><para>
+                    Be self contained, rather than tightly
+                    integrated into the build machine's root
+                    filesystem.
+                    </para></listitem>
+                <listitem><para>
+                    Handle conditional metadata on the target architecture,
+                    operating system, distribution, and machine.
+                    </para></listitem>
+                <listitem><para>
+                    Be easy to use the tools to supply local metadata and packages
+                    against which to operate.
+                    </para></listitem>
+                <listitem><para>
+                    Be easy to use BitBake to collaborate between multiple
+                    projects for their builds.
+                    </para></listitem>
+                <listitem><para>
+                    Provide an inheritance mechanism to share
+                    common metadata between many packages.
+                    </para></listitem>
+            </itemizedlist>
+            Over time it became apparent that some further requirements
+            were necessary:
+            <itemizedlist>
+                <listitem><para>
+                    Handle variants of a base recipe (e.g. native, sdk,
+                    and multilib).
+                    </para></listitem>
+                <listitem><para>
+                    Split metadata into layers and allow layers
+                    to enhance or override other layers.
+                    </para></listitem>
+                <listitem><para>
+                    Allow representation of a given set of input variables
+                    to a task as a checksum.
+                    Based on that checksum, allow acceleration of builds
+                    with prebuilt components.
+                    </para></listitem>
+            </itemizedlist>
+            BitBake satisfies all the original requirements and many more
+            with extensions being made to the basic functionality to
+            reflect the additional requirements.
+            Flexibility and power have always been the priorities.
+            BitBake is highly extensible and supports embedded Python code and
+            execution of any arbitrary tasks.
+        </para>
+    </section>
+
+    <section id="Concepts">
+        <title>Concepts</title>
+
+        <para>
+            BitBake is a program written in the Python language.
+            At the highest level, BitBake interprets metadata, decides
+            what tasks are required to run, and executes those tasks.
+            Similar to GNU Make, BitBake controls how software is
+            built.
+            GNU Make achieves its control through "makefiles", while
+            BitBake uses "recipes".
+        </para>
+
+        <para>
+            BitBake extends the capabilities of a simple
+            tool like GNU Make by allowing for the definition of much more
+            complex tasks, such as assembling entire embedded Linux
+            distributions.
+        </para>
+
+        <para>
+            The remainder of this section introduces several concepts
+            that should be understood in order to better leverage
+            the power of BitBake.
+        </para>
+
+        <section id='recipes'>
+            <title>Recipes</title>
+
+            <para>
+                BitBake Recipes, which are denoted by the file extension
+                <filename>.bb</filename>, are the most basic metadata files.
+                These recipe files provide BitBake with the following:
+                <itemizedlist>
+                    <listitem><para>Descriptive information about the
+                        package (author, homepage, license, and so on)</para></listitem>
+                    <listitem><para>The version of the recipe</para></listitem>
+                    <listitem><para>Existing dependencies (both build
+                        and runtime dependencies)</para></listitem>
+                    <listitem><para>Where the source code resides and
+                        how to fetch it</para></listitem>
+                    <listitem><para>Whether the source code requires
+                        any patches, where to find them, and how to apply
+                        them</para></listitem>
+                    <listitem><para>How to configure and compile the
+                        source code</para></listitem>
+                    <listitem><para>Where on the target machine to install the
+                        package or packages created</para></listitem>
+                </itemizedlist>
+            </para>
+
+            <para>
+                Within the context of BitBake, or any project utilizing BitBake
+                as its build system, files with the <filename>.bb</filename>
+                extension are referred to as recipes.
+                <note>
+                    The term "package" is also commonly used to describe recipes.
+                    However, since the same word is used to describe packaged
+                    output from a project, it is best to maintain a single
+                    descriptive term - "recipes".
+                    Put another way, a single "recipe" file is quite capable
+                    of generating a number of related but separately installable
+                    "packages".
+                    In fact, that ability is fairly common.
+                </note>
+            </para>
+        </section>
+
+        <section id='configuration-files'>
+            <title>Configuration Files</title>
+
+            <para>
+                Configuration files, which are denoted by the
+                <filename>.conf</filename> extension, define
+                various configuration variables that govern the project's build
+                process.
+                These files fall into several areas that define
+                machine configuration options, distribution configuration
+                options, compiler tuning options, general common
+                configuration options, and user configuration options.
+                The main configuration file is the sample
+                <filename>bitbake.conf</filename> file, which is
+                located within the BitBake source tree
+                <filename>conf</filename> directory.
+            </para>
+        </section>
+
+        <section id='classes'>
+            <title>Classes</title>
+
+            <para>
+                Class files, which are denoted by the
+                <filename>.bbclass</filename> extension, contain
+                information that is useful to share between metadata files.
+                The BitBake source tree currently comes with one class metadata file
+                called <filename>base.bbclass</filename>.
+                You can find this file in the
+                <filename>classes</filename> directory.
+                The <filename>base.bbclass</filename> class files is special since it
+                is always included automatically for all recipes
+                and classes.
+                This class contains definitions for standard basic tasks such
+                as fetching, unpacking, configuring (empty by default),
+                compiling (runs any Makefile present), installing (empty by
+                default) and packaging (empty by default).
+                These tasks are often overridden or extended by other classes
+                added during the project development process.
+            </para>
+        </section>
+
+        <section id='layers'>
+            <title>Layers</title>
+
+            <para>
+                Layers allow you to isolate different types of
+                customizations from each other.
+                While you might find it tempting to keep everything in one layer
+                when working on a single project, the more modular you organize
+                your metadata, the easier it is to cope with future changes.
+            </para>
+
+            <para>
+                To illustrate how you can use layers to keep things modular,
+                consider customizations you might make to support a specific target machine.
+                These types of customizations typically reside in a special layer,
+                rather than a general layer, called a Board Support Package (BSP)
+                Layer.
+                Furthermore, the machine customizations should be isolated from
+                recipes and metadata that support a new GUI environment, for
+                example.
+                This situation gives you a couple of layers: one for the machine
+                configurations and one for the GUI environment.
+                It is important to understand, however, that the BSP layer can still
+                make machine-specific additions to recipes within
+                the GUI environment layer without polluting the GUI layer itself
+                with those machine-specific changes.
+                You can accomplish this through a recipe that is a BitBake append
+                (<filename>.bbappend</filename>) file.
+            </para>
+        </section>
+
+        <section id='append-bbappend-files'>
+            <title>Append Files</title>
+
+            <para>
+                Append files, which are files that have the
+                <filename>.bbappend</filename> file extension, extend or
+                override information in an existing recipe file.
+            </para>
+
+            <para>
+                BitBake expects every append file to have a corresponding recipe file.
+                Furthermore, the append file and corresponding recipe file
+                must use the same root filename.
+                The filenames can differ only in the file type suffix used
+                (e.g. <filename>formfactor_0.0.bb</filename> and
+                <filename>formfactor_0.0.bbappend</filename>).
+            </para>
+
+            <para>
+                Information in append files extends or
+                overrides the information in the underlying,
+                similarly-named recipe files.
+            </para>
+
+            <para>
+                When you name an append file, you can use the
+                wildcard character (%) to allow for matching recipe names.
+                For example, suppose you have an append file named
+                as follows:
+                <literallayout class='monospaced'>
+     busybox_1.21.%.bbappend
+                </literallayout>
+                That append file would match any <filename>busybox_1.21.x.bb</filename>
+                version of the recipe.
+                So, the append file would match the following recipe names:
+                <literallayout class='monospaced'>
+     busybox_1.21.1.bb
+     busybox_1.21.2.bb
+     busybox_1.21.3.bb
+                </literallayout>
+                If the <filename>busybox</filename> recipe was updated to
+                <filename>busybox_1.3.0.bb</filename>, the append name would not
+                match.
+                However, if you named the append file
+                <filename>busybox_1.%.bbappend</filename>, then you would have a match.
+            </para>
+
+            <para>
+                In the most general case, you could name the append file something as
+                simple as <filename>busybox_%.bbappend</filename> to be entirely
+                version independent.
+            </para>
+        </section>
+    </section>
+
+    <section id='obtaining-bitbake'>
+        <title>Obtaining BitBake</title>
+
+        <para>
+            You can obtain BitBake several different ways:
+            <itemizedlist>
+                <listitem><para><emphasis>Cloning BitBake:</emphasis>
+                    Using Git to clone the BitBake source code repository
+                    is the recommended method for obtaining BitBake.
+                    Cloning the repository makes it easy to get bug fixes
+                    and have access to stable branches and the master
+                    branch.
+                    Once you have cloned BitBake, you should use
+                    the latest stable
+                    branch for development since the master branch is for
+                    BitBake development and might contain less stable changes.
+                    </para>
+                    <para>You usually need a version of BitBake
+                    that matches the metadata you are using.
+                    The metadata is generally backwards compatible but
+                    not forward compatible.</para>
+                    <para>Here is an example that clones the BitBake repository:
+                    <literallayout class='monospaced'>
+     $ git clone git://git.openembedded.org/bitbake
+                    </literallayout>
+                    This command clones the BitBake Git repository into a
+                    directory called <filename>bitbake</filename>.
+                    Alternatively, you can
+                    designate a directory after the
+                    <filename>git clone</filename> command
+                    if you want to call the new directory something
+                    other than <filename>bitbake</filename>.
+                    Here is an example that names the directory
+                    <filename>bbdev</filename>:
+                    <literallayout class='monospaced'>
+     $ git clone git://git.openembedded.org/bitbake bbdev
+                    </literallayout></para></listitem>
+                <listitem><para><emphasis>Installation using your Distribution
+                    Package Management System:</emphasis>
+                    This method is not
+                    recommended because the BitBake version that is
+                    provided by your distribution, in most cases,
+                    is several
+                    releases behind a snapshot of the BitBake repository.
+                    </para></listitem>
+                <listitem><para><emphasis>Taking a snapshot of BitBake:</emphasis>
+                    Downloading a snapshot of BitBake from the
+                    source code repository gives you access to a known
+                    branch or release of BitBake.
+                    <note>
+                         Cloning the Git repository, as described earlier,
+                         is the preferred method for getting BitBake.
+                         Cloning the repository makes it easier to update as
+                         patches are added to the stable branches.
+                    </note></para>
+                    <para>The following example downloads a snapshot of
+                    BitBake version 1.17.0:
+                    <literallayout class='monospaced'>
+     $ wget http://git.openembedded.org/bitbake/snapshot/bitbake-1.17.0.tar.gz
+     $ tar zxpvf bitbake-1.17.0.tar.gz
+                    </literallayout>
+                    After extraction of the tarball using the tar utility,
+                    you have a directory entitled
+                    <filename>bitbake-1.17.0</filename>.
+                    </para></listitem>
+                <listitem><para><emphasis>Using the BitBake that Comes With Your
+                    Build Checkout:</emphasis>
+                    A final possibility for getting a copy of BitBake is that it
+                    already comes with your checkout of a larger Bitbake-based build
+                    system, such as Poky or Yocto Project.
+                    Rather than manually checking out individual layers and
+                    gluing them together yourself, you can check
+                    out an entire build system.
+                    The checkout will already include a version of BitBake that
+                    has been thoroughly tested for compatibility with the other
+                    components.
+                    For information on how to check out a particular BitBake-based
+                    build system, consult that build system's supporting documentation.
+                    </para></listitem>
+            </itemizedlist>
+        </para>
+    </section>
+
+    <section id="bitbake-user-manual-command">
+        <title>The BitBake Command</title>
+
+        <para>
+            The <filename>bitbake</filename> command is the primary interface
+            to the BitBake tool.
+            This section presents the BitBake command syntax and provides
+            several execution examples.
+        </para>
+
+        <section id='usage-and-syntax'>
+            <title>Usage and syntax</title>
+
+            <para>
+                Following is the usage and syntax for BitBake:
+                <literallayout class='monospaced'>
+     $ bitbake -h
+     Usage: bitbake [options] [recipename/target recipe:do_task ...]
+
+         Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
+         It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
+         will provide the layer, BBFILES and other configuration information.
+
+     Options:
+       --version             show program's version number and exit
+       -h, --help            show this help message and exit
+       -b BUILDFILE, --buildfile=BUILDFILE
+                             Execute tasks from a specific .bb recipe directly.
+                             WARNING: Does not handle any dependencies from other
+                             recipes.
+       -k, --continue        Continue as much as possible after an error. While the
+                             target that failed and anything depending on it cannot
+                             be built, as much as possible will be built before
+                             stopping.
+       -a, --tryaltconfigs   Continue with builds by trying to use alternative
+                             providers where possible.
+       -f, --force           Force the specified targets/task to run (invalidating
+                             any existing stamp file).
+       -c CMD, --cmd=CMD     Specify the task to execute. The exact options
+                             available depend on the metadata. Some examples might
+                             be 'compile' or 'populate_sysroot' or 'listtasks' may
+                             give a list of the tasks available.
+       -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP
+                             Invalidate the stamp for the specified task such as
+                             'compile' and then run the default task for the
+                             specified target(s).
+       -r PREFILE, --read=PREFILE
+                             Read the specified file before bitbake.conf.
+       -R POSTFILE, --postread=POSTFILE
+                             Read the specified file after bitbake.conf.
+       -v, --verbose         Output more log message data to the terminal.
+       -D, --debug           Increase the debug level. You can specify this more
+                             than once.
+       -n, --dry-run         Don't execute, just go through the motions.
+       -S SIGNATURE_HANDLER, --dump-signatures=SIGNATURE_HANDLER
+                             Dump out the signature construction information, with
+                             no task execution. The SIGNATURE_HANDLER parameter is
+                             passed to the handler. Two common values are none and
+                             printdiff but the handler may define more/less. none
+                             means only dump the signature, printdiff means compare
+                             the dumped signature with the cached one.
+       -p, --parse-only      Quit after parsing the BB recipes.
+       -s, --show-versions   Show current and preferred versions of all recipes.
+       -e, --environment     Show the global or per-recipe environment complete
+                             with information about where variables were
+                             set/changed.
+       -g, --graphviz        Save dependency tree information for the specified
+                             targets in the dot syntax.
+       -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
+                             Assume these dependencies don't exist and are already
+                             provided (equivalent to ASSUME_PROVIDED). Useful to
+                             make dependency graphs more appealing
+       -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
+                             Show debug logging for the specified logging domains
+       -P, --profile         Profile the command and save reports.
+       -u UI, --ui=UI        The user interface to use (depexp, goggle, hob, knotty
+                             or ncurses - default knotty).
+       -t SERVERTYPE, --servertype=SERVERTYPE
+                             Choose which server type to use (process or xmlrpc -
+                             default process).
+       --token=XMLRPCTOKEN   Specify the connection token to be used when
+                             connecting to a remote server.
+       --revisions-changed   Set the exit code depending on whether upstream
+                             floating revisions have changed or not.
+       --server-only         Run bitbake without a UI, only starting a server
+                             (cooker) process.
+       -B BIND, --bind=BIND  The name/address for the bitbake server to bind to.
+       --no-setscene         Do not run any setscene tasks. sstate will be ignored
+                             and everything needed, built.
+       --remote-server=REMOTE_SERVER
+                             Connect to the specified server.
+       -m, --kill-server     Terminate the remote server.
+       --observe-only        Connect to a server as an observing-only client.
+       --status-only         Check the status of the remote bitbake server.
+       -w WRITEEVENTLOG, --write-log=WRITEEVENTLOG
+                             Writes the event log of the build to a bitbake event
+                             json file. Use '' (empty string) to assign the name
+                             automatically.
+                </literallayout>
+            </para>
+        </section>
+
+        <section id='bitbake-examples'>
+            <title>Examples</title>
+
+            <para>
+                This section presents some examples showing how to use BitBake.
+            </para>
+
+            <section id='example-executing-a-task-against-a-single-recipe'>
+                <title>Executing a Task Against a Single Recipe</title>
+
+                <para>
+                    Executing tasks for a single recipe file is relatively simple.
+                    You specify the file in question, and BitBake parses
+                    it and executes the specified task.
+                    If you do not specify a task, BitBake executes the default
+                    task, which is "build”.
+                    BitBake obeys inter-task dependencies when doing
+                    so.
+                </para>
+
+                <para>
+                    The following command runs the build task, which is
+                    the default task, on the <filename>foo_1.0.bb</filename>
+                    recipe file:
+                    <literallayout class='monospaced'>
+     $ bitbake -b foo_1.0.bb
+                    </literallayout>
+                    The following command runs the clean task on the
+                    <filename>foo.bb</filename> recipe file:
+                    <literallayout class='monospaced'>
+     $ bitbake -b foo.bb -c clean
+                    </literallayout>
+                    <note>
+                        The "-b" option explicitly does not handle recipe
+                        dependencies.
+                        Other than for debugging purposes, it is instead
+                        recommended that you use the syntax presented in the
+                        next section.
+                    </note>
+                </para>
+            </section>
+
+            <section id='executing-tasks-against-a-set-of-recipe-files'>
+                <title>Executing Tasks Against a Set of Recipe Files</title>
+
+                <para>
+                    There are a number of additional complexities introduced
+                    when one wants to manage multiple <filename>.bb</filename>
+                    files.
+                    Clearly there needs to be a way to tell BitBake what
+                    files are available and, of those, which you
+                    want to execute.
+                    There also needs to be a way for each recipe
+                    to express its dependencies, both for build-time and
+                    runtime.
+                    There must be a way for you to express recipe preferences
+                    when multiple recipes provide the same functionality, or when
+                    there are multiple versions of a recipe.
+                </para>
+
+                <para>
+                    The <filename>bitbake</filename> command, when not using
+                    "--buildfile" or "-b" only accepts a "PROVIDES".
+                    You cannot provide anything else.
+                    By default, a recipe file generally "PROVIDES" its
+                    "packagename" as shown in the following example:
+                    <literallayout class='monospaced'>
+     $ bitbake foo
+                    </literallayout>
+                    This next example "PROVIDES" the package name and also uses
+                    the "-c" option to tell BitBake to just execute the
+                    <filename>do_clean</filename> task:
+                    <literallayout class='monospaced'>
+     $ bitbake -c clean foo
+                    </literallayout>
+                </para>
+            </section>
+
+            <section id='generating-dependency-graphs'>
+                <title>Generating Dependency Graphs</title>
+
+                <para>
+                    BitBake is able to generate dependency graphs using
+                    the <filename>dot</filename> syntax.
+                    You can convert these graphs into images using the
+                    <filename>dot</filename> tool from
+                    <ulink url='http://www.graphviz.org'>Graphviz</ulink>.
+                </para>
+
+                <para>
+                    When you generate a dependency graph, BitBake writes four files
+                    to the current working directory:
+                    <itemizedlist>
+                        <listitem><para><emphasis><filename>package-depends.dot</filename>:</emphasis>
+                            Shows BitBake's knowledge of dependencies between
+                            runtime targets.
+                            </para></listitem>
+                        <listitem><para><emphasis><filename>pn-depends.dot</filename>:</emphasis>
+                            Shows dependencies between build-time targets
+                            (i.e. recipes).
+                            </para></listitem>
+                        <listitem><para><emphasis><filename>task-depends.dot</filename>:</emphasis>
+                            Shows dependencies between tasks.
+                            </para></listitem>
+                        <listitem><para><emphasis><filename>pn-buildlist</filename>:</emphasis>
+                            Shows a simple list of targets that are to be built.
+                            </para></listitem>
+                    </itemizedlist>
+                </para>
+
+                <para>
+                    To stop depending on common depends, use the "-I" depend
+                    option and BitBake omits them from the graph.
+                    Leaving this information out can produce more readable graphs.
+                    This way, you can remove from the graph
+                    <filename>DEPENDS</filename> from inherited classes
+                    such as <filename>base.bbclass</filename>.
+                </para>
+
+                <para>
+                    Here are two examples that create dependency graphs.
+                    The second example omits depends common in OpenEmbedded from
+                    the graph:
+                    <literallayout class='monospaced'>
+     $ bitbake -g foo
+
+     $ bitbake -g -I virtual/kernel -I eglibc foo
+                    </literallayout>
+                </para>
+            </section>
+        </section>
+    </section>
+</chapter>

bitbake/doc/bitbake-user-manual/bitbake-user-manual-style.css

@@ -0,0 +1,984 @@
+/*
+   Generic XHTML / DocBook XHTML CSS Stylesheet.
+
+   Browser wrangling and typographic design by
+      Oyvind Kolas / pippin@gimp.org
+
+   Customised for Poky by
+      Matthew Allum / mallum@o-hand.com
+
+   Thanks to:
+     Liam R. E. Quin
+     William Skaggs
+     Jakub Steiner
+
+   Structure
+   ---------
+
+   The stylesheet is divided into the following sections:
+
+       Positioning
+          Margins, paddings, width, font-size, clearing.
+       Decorations
+          Borders, style
+       Colors
+          Colors
+       Graphics
+          Graphical backgrounds
+       Nasty IE tweaks
+          Workarounds needed to make it work in internet explorer,
+          currently makes the stylesheet non validating, but up until
+          this point it is validating.
+       Mozilla extensions
+          Transparency for footer
+	  Rounded corners on boxes
+
+*/
+
+
+  /*************** /
+ /  Positioning   /
+/ ***************/
+
+body {
+  font-family: Verdana, Sans, sans-serif;
+
+  min-width: 640px;
+  width: 80%;
+  margin:  0em auto;
+  padding: 2em 5em 5em 5em;
+  color: #333;
+}
+
+h1,h2,h3,h4,h5,h6,h7 {
+  font-family: Arial, Sans;
+  color: #00557D;
+  clear: both;
+}
+
+h1 {
+  font-size: 2em;
+  text-align: left;
+  padding: 0em 0em 0em 0em;
+  margin: 2em 0em 0em 0em;
+}
+
+h2.subtitle {
+  margin: 0.10em 0em 3.0em 0em;
+  padding: 0em 0em 0em 0em;
+  font-size: 1.8em;
+  padding-left: 20%;
+  font-weight: normal;
+  font-style: italic;
+}
+
+h2 {
+  margin: 2em 0em 0.66em 0em;
+  padding: 0.5em 0em 0em 0em;
+  font-size: 1.5em;
+  font-weight: bold;
+}
+
+h3.subtitle {
+  margin: 0em 0em 1em 0em;
+  padding: 0em 0em 0em 0em;
+  font-size: 142.14%;
+  text-align: right;
+}
+
+h3 {
+  margin: 1em 0em 0.5em 0em;
+  padding: 1em 0em 0em 0em;
+  font-size: 140%;
+  font-weight: bold;
+}
+
+h4 {
+  margin: 1em 0em 0.5em 0em;
+  padding: 1em 0em 0em 0em;
+  font-size: 120%;
+  font-weight: bold;
+}
+
+h5 {
+  margin: 1em 0em 0.5em 0em;
+  padding: 1em 0em 0em 0em;
+  font-size: 110%;
+  font-weight: bold;
+}
+
+h6 {
+  margin: 1em 0em 0em 0em;
+  padding: 1em 0em 0em 0em;
+  font-size: 110%;
+  font-weight: bold;
+}
+
+.authorgroup {
+  background-color: transparent;
+  background-repeat: no-repeat;
+  padding-top: 256px;
+  background-image: url("figures/bitbake-title.png");
+  background-position: left top;
+  margin-top: -256px;
+  padding-right: 50px;
+  margin-left: 0px;
+  text-align: right;
+  width: 740px;
+}
+
+h3.author {
+  margin: 0em 0me 0em 0em;
+  padding: 0em 0em 0em 0em;
+  font-weight: normal;
+  font-size: 100%;
+  color: #333;
+  clear: both;
+}
+
+.author tt.email {
+  font-size: 66%;
+}
+
+.titlepage hr {
+  width: 0em;
+  clear: both;
+}
+
+.revhistory {
+  padding-top: 2em;
+  clear: both;
+}
+
+.toc,
+.list-of-tables,
+.list-of-examples,
+.list-of-figures {
+  padding: 1.33em 0em 2.5em 0em;
+  color: #00557D;
+}
+
+.toc p,
+.list-of-tables p,
+.list-of-figures p,
+.list-of-examples p {
+  padding: 0em 0em 0em 0em;
+  padding: 0em 0em 0.3em;
+  margin: 1.5em 0em 0em 0em;
+}
+
+.toc p b,
+.list-of-tables p b,
+.list-of-figures p b,
+.list-of-examples p b{
+  font-size: 100.0%;
+  font-weight: bold;
+}
+
+.toc dl,
+.list-of-tables dl,
+.list-of-figures dl,
+.list-of-examples dl {
+  margin: 0em 0em 0.5em 0em;
+  padding: 0em 0em 0em 0em;
+}
+
+.toc dt {
+  margin: 0em 0em 0em 0em;
+  padding: 0em 0em 0em 0em;
+}
+
+.toc dd {
+  margin: 0em 0em 0em 2.6em;
+  padding: 0em 0em 0em 0em;
+}
+
+div.glossary dl,
+div.variablelist dl {
+}
+
+.glossary dl dt,
+.variablelist dl dt,
+.variablelist dl dt span.term {
+  font-weight: normal;
+  width: 20em;
+  text-align: right;
+}
+
+.variablelist dl dt {
+  margin-top: 0.5em;
+}
+
+.glossary dl dd,
+.variablelist dl dd {
+  margin-top: -1em;
+  margin-left: 25.5em;
+}
+
+.glossary dd p,
+.variablelist dd p {
+  margin-top: 0em;
+  margin-bottom: 1em;
+}
+
+
+div.calloutlist table td {
+  padding: 0em 0em 0em 0em;
+  margin: 0em 0em 0em 0em;
+}
+
+div.calloutlist table td p {
+  margin-top: 0em;
+  margin-bottom: 1em;
+}
+
+div p.copyright {
+  text-align: left;
+}
+
+div.legalnotice p.legalnotice-title {
+  margin-bottom: 0em;
+}
+
+p {
+  line-height: 1.5em;
+  margin-top: 0em;
+
+}
+
+dl {
+  padding-top: 0em;
+}
+
+hr {
+  border: solid 1px;
+}
+
+
+.mediaobject,
+.mediaobjectco {
+  text-align: center;
+}
+
+img {
+  border: none;
+}
+
+ul {
+  padding: 0em 0em 0em 1.5em;
+}
+
+ul li {
+  padding: 0em 0em 0em 0em;
+}
+
+ul li p {
+  text-align: left;
+}
+
+table {
+  width :100%;
+}
+
+th {
+  padding: 0.25em;
+  text-align: left;
+  font-weight: normal;
+  vertical-align: top;
+}
+
+td {
+  padding: 0.25em;
+  vertical-align: top;
+}
+
+p a[id] {
+  margin: 0px;
+  padding: 0px;
+  display: inline;
+  background-image: none;
+}
+
+a {
+  text-decoration: underline;
+  color: #444;
+}
+
+pre {
+    overflow: auto;
+}
+
+a:hover {
+  text-decoration: underline;
+  /*font-weight: bold;*/
+}
+
+/* This style defines how the permalink character
+   appears by itself and when hovered over with
+   the mouse. */
+
+[alt='Permalink'] { color: #eee; }
+[alt='Permalink']:hover { color: black; }
+
+
+div.informalfigure,
+div.informalexample,
+div.informaltable,
+div.figure,
+div.table,
+div.example {
+  margin: 1em 0em;
+  padding: 1em;
+  page-break-inside: avoid;
+}
+
+
+div.informalfigure p.title b,
+div.informalexample p.title b,
+div.informaltable p.title b,
+div.figure p.title b,
+div.example p.title b,
+div.table p.title b{
+    padding-top: 0em;
+    margin-top: 0em;
+    font-size: 100%;
+    font-weight: normal;
+}
+
+.mediaobject .caption,
+.mediaobject .caption p  {
+  text-align: center;
+  font-size: 80%;
+  padding-top: 0.5em;
+  padding-bottom: 0.5em;
+}
+
+.epigraph {
+  padding-left: 55%;
+  margin-bottom: 1em;
+}
+
+.epigraph p {
+  text-align: left;
+}
+
+.epigraph .quote {
+  font-style: italic;
+}
+.epigraph .attribution {
+  font-style: normal;
+  text-align: right;
+}
+
+span.application {
+  font-style: italic;
+}
+
+.programlisting {
+  font-family: monospace;
+  font-size: 80%;
+  white-space: pre;
+  margin: 1.33em 0em;
+  padding: 1.33em;
+}
+
+.tip,
+.warning,
+.caution,
+.note {
+  margin-top: 1em;
+  margin-bottom: 1em;
+
+}
+
+/* force full width of table within div */
+.tip table,
+.warning table,
+.caution table,
+.note table {
+  border: none;
+  width: 100%;
+}
+
+
+.tip table th,
+.warning table th,
+.caution table th,
+.note table th {
+  padding: 0.8em 0.0em 0.0em 0.0em;
+  margin : 0em 0em 0em 0em;
+}
+
+.tip p,
+.warning p,
+.caution p,
+.note p {
+  margin-top: 0.5em;
+  margin-bottom: 0.5em;
+  padding-right: 1em;
+  text-align: left;
+}
+
+.acronym {
+  text-transform: uppercase;
+}
+
+b.keycap,
+.keycap {
+  padding: 0.09em 0.3em;
+  margin: 0em;
+}
+
+.itemizedlist li {
+  clear: none;
+}
+
+.filename {
+  font-size: medium;
+  font-family: Courier, monospace;
+}
+
+
+div.navheader, div.heading{
+  position: absolute;
+  left: 0em;
+  top: 0em;
+  width: 100%;
+  background-color: #cdf;
+  width: 100%;
+}
+
+div.navfooter, div.footing{
+  position: fixed;
+  left: 0em;
+  bottom: 0em;
+  background-color: #eee;
+  width: 100%;
+}
+
+
+div.navheader td,
+div.navfooter td {
+  font-size: 66%;
+}
+
+div.navheader table th {
+  /*font-family: Georgia, Times, serif;*/
+  /*font-size: x-large;*/
+  font-size: 80%;
+}
+
+div.navheader table {
+  border-left: 0em;
+  border-right: 0em;
+  border-top: 0em;
+  width: 100%;
+}
+
+div.navfooter table {
+  border-left: 0em;
+  border-right: 0em;
+  border-bottom: 0em;
+  width: 100%;
+}
+
+div.navheader table td a,
+div.navfooter table td a {
+  color: #777;
+  text-decoration: none;
+}
+
+/* normal text in the footer */
+div.navfooter table td {
+  color: black;
+}
+
+div.navheader table td a:visited,
+div.navfooter table td a:visited {
+  color: #444;
+}
+
+
+/* links in header and footer */
+div.navheader table td a:hover,
+div.navfooter table td a:hover {
+  text-decoration: underline;
+  background-color: transparent;
+  color: #33a;
+}
+
+div.navheader hr,
+div.navfooter hr {
+  display: none;
+}
+
+
+.qandaset tr.question td p {
+  margin: 0em 0em 1em 0em;
+  padding: 0em 0em 0em 0em;
+}
+
+.qandaset tr.answer td p {
+  margin: 0em 0em 1em 0em;
+  padding: 0em 0em 0em 0em;
+}
+.answer td {
+  padding-bottom: 1.5em;
+}
+
+.emphasis {
+  font-weight: bold;
+}
+
+
+  /************* /
+ / decorations  /
+/ *************/
+
+.titlepage {
+}
+
+.part .title {
+}
+
+.subtitle {
+    border: none;
+}
+
+/*
+h1 {
+  border: none;
+}
+
+h2 {
+  border-top: solid 0.2em;
+  border-bottom: solid 0.06em;
+}
+
+h3 {
+  border-top: 0em;
+  border-bottom: solid 0.06em;
+}
+
+h4 {
+  border: 0em;
+  border-bottom: solid 0.06em;
+}
+
+h5 {
+  border: 0em;
+}
+*/
+
+.programlisting {
+  border: solid 1px;
+}
+
+div.figure,
+div.table,
+div.informalfigure,
+div.informaltable,
+div.informalexample,
+div.example {
+  border: 1px solid;
+}
+
+
+
+.tip,
+.warning,
+.caution,
+.note {
+  border: 1px solid;
+}
+
+.tip table th,
+.warning table th,
+.caution table th,
+.note table th {
+  border-bottom: 1px solid;
+}
+
+.question td {
+  border-top: 1px solid black;
+}
+
+.answer {
+}
+
+
+b.keycap,
+.keycap {
+  border: 1px solid;
+}
+
+
+div.navheader, div.heading{
+  border-bottom: 1px solid;
+}
+
+
+div.navfooter, div.footing{
+  border-top: 1px solid;
+}
+
+  /********* /
+ /  colors  /
+/ *********/
+
+body {
+  color: #333;
+  background: white;
+}
+
+a {
+  background: transparent;
+}
+
+a:hover {
+  background-color: #dedede;
+}
+
+
+h1,
+h2,
+h3,
+h4,
+h5,
+h6,
+h7,
+h8 {
+  background-color: transparent;
+}
+
+hr {
+  border-color: #aaa;
+}
+
+
+.tip, .warning, .caution, .note {
+  border-color: #fff;
+}
+
+
+.tip table th,
+.warning table th,
+.caution table th,
+.note table th {
+  border-bottom-color: #fff;
+}
+
+
+.warning {
+  background-color: #f0f0f2;
+}
+
+.caution {
+  background-color: #f0f0f2;
+}
+
+.tip {
+  background-color: #f0f0f2;
+}
+
+.note {
+  background-color: #f0f0f2;
+}
+
+.glossary dl dt,
+.variablelist dl dt,
+.variablelist dl dt span.term {
+  color: #044;
+}
+
+div.figure,
+div.table,
+div.example,
+div.informalfigure,
+div.informaltable,
+div.informalexample {
+  border-color: #aaa;
+}
+
+pre.programlisting {
+  color: black;
+  background-color: #fff;
+  border-color: #aaa;
+  border-width: 2px;
+}
+
+.guimenu,
+.guilabel,
+.guimenuitem {
+  background-color: #eee;
+}
+
+
+b.keycap,
+.keycap {
+  background-color: #eee;
+  border-color: #999;
+}
+
+
+div.navheader {
+  border-color: black;
+}
+
+
+div.navfooter {
+  border-color: black;
+}
+
+
+  /*********** /
+ /  graphics  /
+/ ***********/
+
+/*
+body {
+  background-image: url("images/body_bg.jpg");
+  background-attachment: fixed;
+}
+
+.navheader,
+.note,
+.tip {
+  background-image: url("images/note_bg.jpg");
+  background-attachment: fixed;
+}
+
+.warning,
+.caution {
+  background-image: url("images/warning_bg.jpg");
+  background-attachment: fixed;
+}
+
+.figure,
+.informalfigure,
+.example,
+.informalexample,
+.table,
+.informaltable {
+  background-image: url("images/figure_bg.jpg");
+  background-attachment: fixed;
+}
+
+*/
+h1,
+h2,
+h3,
+h4,
+h5,
+h6,
+h7{
+}
+
+/*
+Example of how to stick an image as part of the title.
+
+div.article .titlepage .title
+{
+  background-image: url("figures/white-on-black.png");
+  background-position: center;
+  background-repeat: repeat-x;
+}
+*/
+
+div.preface .titlepage .title,
+div.colophon .title,
+div.chapter .titlepage .title,
+div.article .titlepage .title
+{
+}
+
+div.section div.section .titlepage .title,
+div.sect2 .titlepage .title {
+    background: none;
+}
+
+
+h1.title {
+  background-color: transparent;
+  background-repeat: no-repeat;
+  height: 256px;
+  text-indent: -9000px;
+  overflow:hidden;
+}
+
+h2.subtitle {
+  background-color: transparent;
+  text-indent: -9000px;
+  overflow:hidden;
+  width: 0px;
+  display: none;
+}
+
+  /*************************************** /
+ /  pippin.gimp.org specific alterations  /
+/ ***************************************/
+
+/*
+div.heading, div.navheader {
+  color: #777;
+  font-size: 80%;
+  padding: 0;
+  margin: 0;
+  text-align: left;
+  position: absolute;
+  top: 0px;
+  left: 0px;
+  width: 100%;
+  height: 50px;
+  background: url('/gfx/heading_bg.png') transparent;
+  background-repeat: repeat-x;
+  background-attachment: fixed;
+  border: none;
+}
+
+div.heading a {
+  color: #444;
+}
+
+div.footing, div.navfooter {
+  border: none;
+  color: #ddd;
+  font-size: 80%;
+  text-align:right;
+
+  width: 100%;
+  padding-top: 10px;
+  position: absolute;
+  bottom: 0px;
+  left: 0px;
+
+  background: url('/gfx/footing_bg.png') transparent;
+}
+*/
+
+
+
+  /****************** /
+ /  nasty ie tweaks  /
+/ ******************/
+
+/*
+div.heading, div.navheader {
+  width:expression(document.body.clientWidth + "px");
+}
+
+div.footing, div.navfooter {
+  width:expression(document.body.clientWidth + "px");
+  margin-left:expression("-5em");
+}
+body {
+  padding:expression("4em 5em 0em 5em");
+}
+*/
+
+  /**************************************** /
+ / mozilla vendor specific css extensions  /
+/ ****************************************/
+/*
+div.navfooter, div.footing{
+  -moz-opacity: 0.8em;
+}
+
+div.figure,
+div.table,
+div.informalfigure,
+div.informaltable,
+div.informalexample,
+div.example,
+.tip,
+.warning,
+.caution,
+.note {
+  -moz-border-radius: 0.5em;
+}
+
+b.keycap,
+.keycap {
+  -moz-border-radius: 0.3em;
+}
+*/
+
+table tr td table tr td {
+  display: none;
+}
+
+
+hr {
+  display: none;
+}
+
+table {
+  border: 0em;
+}
+
+ .photo {
+  float: right;
+  margin-left:   1.5em;
+  margin-bottom: 1.5em;
+  margin-top: 0em;
+  max-width:      17em;
+  border:     1px solid gray;
+  padding:    3px;
+  background: white;
+}
+ .seperator {
+   padding-top: 2em;
+   clear: both;
+  }
+
+  #validators {
+      margin-top: 5em;
+      text-align: right;
+      color: #777;
+  }
+  @media print {
+      body {
+          font-size: 8pt;
+      }
+      .noprint {
+          display: none;
+      }
+  }
+
+
+.tip,
+.note {
+   background: #f0f0f2;
+   color: #333;
+   padding: 20px;
+   margin: 20px;
+}
+
+.tip h3,
+.note h3 {
+   padding: 0em;
+   margin: 0em;
+   font-size: 2em;
+   font-weight: bold;
+   color: #333;
+}
+
+.tip a,
+.note a {
+   color: #333;
+   text-decoration: underline;
+}
+
+.footnote {
+   font-size: small;
+   color: #333;
+}
+
+/* Changes the announcement text */
+.tip h3,
+.warning h3,
+.caution h3,
+.note h3 {
+   font-size:large;
+   color: #00557D;
+}

bitbake/doc/bitbake-user-manual/bitbake-user-manual.xml

@@ -0,0 +1,88 @@
+<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+
+<book id='bitbake-user-manual' lang='en'
+        xmlns:xi="http://www.w3.org/2003/XInclude"
+        xmlns="http://docbook.org/ns/docbook"
+        >
+    <bookinfo>
+
+        <mediaobject>
+            <imageobject>
+                <imagedata fileref='figures/bitbake-title.png'
+                    format='SVG'
+                    align='left' scalefit='1' width='100%'/>
+            </imageobject>
+        </mediaobject>
+
+        <title>
+            BitBake User Manual
+        </title>
+
+        <authorgroup>
+            <author>
+                <firstname>Richard Purdie, Chris Larson, and </firstname> <surname>Phil Blundell</surname>
+                <affiliation>
+                    <orgname>BitBake Community</orgname>
+                </affiliation>
+                <email>bitbake-devel@lists.openembedded.org</email>
+            </author>
+        </authorgroup>
+
+<!--
+# Add in some revision history if we want it here.
+        <revhistory>
+            <revision>
+                <revnumber>x.x</revnumber>
+                <date>dd month year</date>
+                <revremark>Some relevent comment</revremark>
+            </revision>
+            <revision>
+                <revnumber>x.x</revnumber>
+                <date>dd month year</date>
+                <revremark>Some relevent comment</revremark>
+            </revision>
+            <revision>
+                <revnumber>x.x</revnumber>
+                <date>dd month year</date>
+                <revremark>Some relevent comment</revremark>
+            </revision>
+            <revision>
+                <revnumber>x.x</revnumber>
+                <date>dd month year</date>
+                <revremark>Some relevent comment</revremark>
+            </revision>
+       </revhistory>
+-->
+
+        <copyright>
+            <year>2004-2016</year>
+            <holder>Richard Purdie</holder>
+            <holder>Chris Larson</holder>
+            <holder>and Phil Blundell</holder>
+        </copyright>
+
+        <legalnotice>
+            <para>
+                This work is licensed under the Creative Commons Attribution License.
+                To view a copy of this license, visit
+                <ulink url="http://creativecommons.org/licenses/by/2.5/">http://creativecommons.org/licenses/by/2.5/</ulink>
+                or send a letter to Creative Commons, 444 Castro Street,
+                Suite 900, Mountain View, California 94041, USA.
+            </para>
+        </legalnotice>
+    </bookinfo>
+
+    <xi:include href="bitbake-user-manual-intro.xml"/>
+
+    <xi:include href="bitbake-user-manual-execution.xml"/>
+
+    <xi:include href="bitbake-user-manual-metadata.xml"/>
+
+    <xi:include href="bitbake-user-manual-fetching.xml"/>
+
+    <xi:include href="bitbake-user-manual-ref-variables.xml"/>
+
+    <xi:include href="bitbake-user-manual-hello.xml"/>
+
+</book>

bitbake/doc/bitbake-user-manual/html.css

@@ -0,0 +1,281 @@
+/* Feuille de style DocBook du projet Traduc.org                */
+/* DocBook CSS stylesheet of the Traduc.org project             */
+
+/* (c) Jean-Philippe Gu<47>rard - 14 ao<61>t 2004                     */
+/* (c) Jean-Philippe Gu<47>rard - 14 August 2004                   */
+
+/* Cette feuille de style est libre, vous pouvez la             */
+/* redistribuer et la modifier selon les termes de la Licence   */
+/* Art Libre. Vous trouverez un exemplaire de cette Licence sur */
+/* http://tigreraye.org/Petit-guide-du-traducteur.html#licence-art-libre */
+
+/* This work of art is free, you can redistribute it and/or     */
+/* modify it according to terms of the Free Art license. You    */
+/* will find a specimen of this license on the Copyleft         */
+/* Attitude web site: http://artlibre.org as well as on other   */
+/* sites.                                                       */
+/* Please note that the French version of this licence as shown */
+/* on http://tigreraye.org/Petit-guide-du-traducteur.html#licence-art-libre */
+/* is only official licence of this document. The English       */
+/* is only provided to help you understand this licence.        */
+
+/* La derni<6E>re version de cette feuille de style est toujours   */
+/* disponible sur<75>: http://tigreraye.org/style.css              */
+/* Elle est <20>galement disponible sur<75>:                          */
+/* http://www.traduc.org/docs/HOWTO/lecture/style.css           */
+
+/* The latest version of this stylesheet is available from:     */
+/* http://tigreraye.org/style.css                               */
+/* It is also available on:                                     */
+/* http://www.traduc.org/docs/HOWTO/lecture/style.css           */
+
+/* N'h<>sitez pas <20> envoyer vos commentaires et corrections <20>    */
+/* Jean-Philippe Gu<47>rard <jean-philippe.guerard@tigreraye.org>  */
+
+/* Please send feedback and bug reports to                      */
+/* Jean-Philippe Gu<47>rard <jean-philippe.guerard@tigreraye.org>  */
+
+/* $Id: style.css,v 1.14 2004/09/10 20:12:09 fevrier Exp fevrier $ */
+
+/* Pr<50>sentation g<>n<EFBFBD>rale du document */
+/* Overall document presentation */
+
+body {
+    /*
+    font-family: Apolline, "URW Palladio L", Garamond, jGaramond,
+                 "Bitstream Cyberbit", "Palatino Linotype", serif;
+     */
+    margin: 7%;
+    background-color: white;
+}
+
+/* Taille du texte */
+/* Text size */
+
+* { font-size: 100%; }
+
+/* Gestion des textes mis en relief imbriqu<71>s */
+/* Embedded emphasis */
+
+em { font-style: italic; }
+em em { font-style: normal; }
+em em em { font-style: italic; }
+
+/* Titres */
+/* Titles */
+
+h1 { font-size: 200%; font-weight: 900; }
+h2 { font-size: 160%; font-weight: 900; }
+h3 { font-size: 130%; font-weight: bold; }
+h4 { font-size: 115%; font-weight: bold; }
+h5 { font-size: 108%; font-weight: bold; }
+h6 {                  font-weight: bold; }
+
+/* Nom de famille en petites majuscules (uniquement en fran<61>ais) */
+/* Last names in small caps (for French only) */
+
+*[class~="surname"]:lang(fr) { font-variant: small-caps; }
+
+/* Blocs de citation */
+/* Quotation blocs */
+
+div[class~="blockquote"] {
+  border: solid 2px #AAA;
+  padding: 5px;
+  margin: 5px;
+}
+
+div[class~="blockquote"] > table {
+  border: none;
+}
+
+/* Blocs lit<69>raux<75>: fond gris clair */
+/* Literal blocs: light gray background */
+
+*[class~="literallayout"] {
+  background: #f0f0f0;
+  padding: 5px;
+  margin: 5px;
+}
+
+/* Programmes et captures texte<74>: fond bleu clair */
+/* Listing and text screen snapshots: light blue background */
+
+*[class~="programlisting"], *[class~="screen"] {
+  background: #f0f0ff;
+  padding: 5px;
+  margin: 5px;
+}
+
+/* Les textes <20> remplacer sont surlign<67>s en vert p<>le */
+/* Replaceable text in highlighted in pale green */
+
+*[class~="replaceable"] { 
+    background-color: #98fb98;
+    font-style: normal; }
+
+/* Tables<65>: fonds gris clair & bords simples */
+/* Tables: light gray background and solid borders */
+
+*[class~="table"] *[class~="title"] { width:100%; border: 0px; }
+
+table {
+    border: 1px solid #aaa;
+    border-collapse: collapse;
+    padding: 2px;
+    margin: 5px;
+}
+
+/* Listes simples en style table */
+/* Simples lists in table presentation */
+
+table[class~="simplelist"] {
+    background-color: #F0F0F0;
+    margin: 5px;
+    border: solid 1px #AAA;
+}
+
+table[class~="simplelist"] td {
+    border: solid 1px #AAA;
+}
+
+/* Les tables */
+/* Tables */
+
+*[class~="table"] table {
+    background-color: #F0F0F0;
+    border: solid 1px #AAA;
+}
+*[class~="informaltable"] table { background-color: #F0F0F0; }
+
+th,td {
+    vertical-align: baseline;
+    text-align: left;
+    padding: 0.1em 0.3em;
+    empty-cells: show; 
+}
+
+/* Alignement des colonnes */
+/* Colunms alignment */
+
+td[align=center] ,  th[align=center]  { text-align: center; }
+td[align=right] ,   th[align=right]   { text-align: right; }
+td[align=left] ,    th[align=left]    { text-align: left; }
+td[align=justify] , th[align=justify] { text-align: justify; }
+
+/* Pas de marge autour des images */
+/* No inside margins for images */
+
+img { border: 0; }
+
+/* Les liens ne sont pas soulign<67>s */
+/* No underlines for links */
+
+:link , :visited , :active { text-decoration: none; }
+
+/* Prudence<63>: cadre jaune et fond jaune clair */
+/* Caution: yellow border and light yellow background */
+
+*[class~="caution"] {
+    border: solid 2px yellow;
+    background-color: #ffffe0;
+    padding: 1em 6px 1em ;
+    margin: 5px;
+}
+
+*[class~="caution"] th {
+    vertical-align: middle
+}
+
+*[class~="caution"] table {
+    background-color: #ffffe0;
+    border: none;
+}
+
+/* Note importante<74>: cadre jaune et fond jaune clair */
+/* Important: yellow border and light yellow background */
+
+*[class~="important"] {
+    border: solid 2px yellow;
+    background-color: #ffffe0;
+    padding: 1em 6px 1em;
+    margin: 5px;
+}
+
+*[class~="important"] th {
+    vertical-align: middle
+}
+
+*[class~="important"] table  {
+    background-color: #ffffe0;
+    border: none;
+}
+
+/* Mise en <20>vidence<63>: texte l<>g<EFBFBD>rement plus grand */
+/* Highlights: slightly larger texts */
+
+*[class~="highlights"] {
+    font-size:  110%;
+}
+
+/* Note<74>: cadre bleu et fond bleu clair */
+/* Notes: blue border and light blue background */
+
+*[class~="note"]   {
+    border: solid 2px #7099C5;
+    background-color: #f0f0ff;
+    padding: 1em 6px 1em ;
+    margin: 5px;
+}
+
+*[class~="note"] th {
+    vertical-align: middle
+}
+
+*[class~="note"] table {
+    background-color: #f0f0ff;
+    border: none;
+}
+
+/* Astuce<63>: cadre vert et fond vert clair */
+/* Tip: green border and light green background */
+
+*[class~="tip"] {
+    border: solid 2px #00ff00;
+    background-color: #f0ffff;
+    padding: 1em 6px 1em ;
+    margin: 5px;
+}
+
+*[class~="tip"] th {
+    vertical-align: middle;
+}
+
+*[class~="tip"] table {
+    background-color: #f0ffff;
+    border: none;
+}
+
+/* Avertissement<6E>: cadre rouge et fond rouge clair */
+/* Warning: red border and light red background */
+
+*[class~="warning"] {
+    border: solid 2px #ff0000;
+    background-color: #fff0f0; 
+    padding: 1em 6px 1em ;
+    margin: 5px;
+}
+
+*[class~="warning"] th {
+    vertical-align: middle;
+}
+                    
+
+*[class~="warning"] table {
+    background-color: #fff0f0;
+    border: none;
+}
+
+/* Fin */
+/* The End */
+

bitbake/doc/bitbake.1

@@ -0,0 +1,142 @@
+.\"                                      Hey, EMACS: -*- nroff -*-
+.\" First parameter, NAME, should be all caps
+.\" Second parameter, SECTION, should be 1-8, maybe w/ subsection
+.\" other parameters are allowed: see man(7), man(1)
+.TH BITBAKE 1 "November 19, 2006"
+.\" Please adjust this date whenever revising the manpage.
+.\"
+.\" Some roff macros, for reference:
+.\" .nh        disable hyphenation
+.\" .hy        enable hyphenation
+.\" .ad l      left justify
+.\" .ad b      justify to both left and right margins
+.\" .nf        disable filling
+.\" .fi        enable filling
+.\" .br        insert line break
+.\" .sp <n>    insert n+1 empty lines
+.\" for manpage-specific macros, see man(7)
+.SH NAME
+BitBake \- simple tool for the execution of tasks
+.SH SYNOPSIS
+.B bitbake
+.RI [ options ] " packagenames"
+.br
+.SH DESCRIPTION
+This manual page documents briefly the
+.B bitbake
+command.
+.PP
+.\" TeX users may be more comfortable with the \fB<whatever>\fP and
+.\" \fI<whatever>\fP escape sequences to invode bold face and italics, 
+.\" respectively.
+\fBbitbake\fP is a program that executes the specified task (default is 'build')
+for a given set of BitBake files.
+.br
+It expects that BBFILES is defined, which is a space separated list of files to
+be executed. BBFILES does support wildcards.
+.br
+Default BBFILES are the .bb files in the current directory.
+.SH OPTIONS
+This program follow the usual GNU command line syntax, with long
+options starting with two dashes (`-').
+.TP
+.B \-h, \-\-help
+Show summary of options.
+.TP
+.B \-\-version
+Show version of program.
+.TP
+.B \-bBUILDFILE, \-\-buildfile=BUILDFILE
+execute the task against this .bb file, rather than a package from BBFILES.
+.TP
+.B \-k, \-\-continue
+continue as much as possible after an error. While the target that failed, and
+those that depend on it, cannot be remade, the other dependencies of these
+targets can be processed all the same.
+.TP
+.B \-a, \-\-tryaltconfigs
+continue with builds by trying to use alternative providers where possible.
+.TP
+.B \-f, \-\-force
+force run of specified cmd, regardless of stamp status
+.TP
+.B \-i, \-\-interactive
+drop into the interactive mode also called the BitBake shell.
+.TP
+.B \-cCMD, \-\-cmd=CMD
+Specify task to execute. Note that this only executes the specified task for
+the providee and the packages it depends on, i.e. 'compile' does not implicitly
+call stage for the dependencies (IOW: use only if you know what you are doing).
+Depending on the base.bbclass a listtasks task is defined and will show
+available tasks.
+.TP
+.B \-rFILE, \-\-read=FILE 
+read the specified file before bitbake.conf
+.TP
+.B \-v, \-\-verbose
+output more chit-chat to the terminal
+.TP
+.B \-D, \-\-debug
+Increase the debug level. You can specify this more than once.
+.TP
+.B \-n, \-\-dry-run
+don't execute, just go through the motions
+.TP
+.B \-p, \-\-parse-only
+quit after parsing the BB files (developers only)
+.TP
+.B \-s, \-\-show-versions
+show current and preferred versions of all packages
+.TP
+.B \-e, \-\-environment
+show the global or per-recipe environment (this is what used to be bbread)
+.TP
+.B \-g, \-\-graphviz
+emit the dependency trees of the specified packages in the dot syntax
+.TP
+.B \-IIGNORED\_DOT\_DEPS, \-\-ignore-deps=IGNORED_DOT_DEPS
+Stop processing at the given list of dependencies when generating dependency
+graphs. This can help to make the graph more appealing
+.TP
+.B \-lDEBUG_DOMAINS, \-\-log-domains=DEBUG_DOMAINS
+Show debug logging for the specified logging domains
+.TP
+.B \-P, \-\-profile
+profile the command and print a report
+.TP
+.B \-uUI, \-\-ui=UI
+User interface to use. Currently, hob, depexp, goggle or ncurses can be specified as UI.
+.TP
+.B \-tSERVERTYPE, \-\-servertype=SERVERTYPE
+Choose which server to use, none, process or xmlrpc.
+.TP
+.B \-\-revisions-changed
+Set the exit code depending on whether upstream floating revisions have changed or not.
+.TP
+.B \-\-server-only
+Run bitbake without UI,  the frontend can connect with bitbake server itself.
+.TP
+.B \-BBIND, \-\-bind=BIND
+The name/address for the bitbake server to bind to.
+.TP
+.B \-\-no\-setscene
+Do not run any setscene tasks, forces builds.
+
+.SH ENVIRONMENT VARIABLES
+bitbake uses the following environment variables to control its
+operation:
+.TP
+.B BITBAKE_UI
+The bitbake user interface; overridden by the \fB-u\fP commandline option.
+
+.SH AUTHORS
+BitBake was written by 
+Phil Blundell,
+Holger Freyther,
+Chris Larson,
+Mickey Lauer,
+Richard Purdie,
+Holger Schurig
+.PP
+This manual page was written by Marcin Juszkiewicz <marcin@hrw.one.pl>
+for the Debian project (but may be used by others).

bitbake/doc/poky.ent

@@ -0,0 +1,59 @@
+<!ENTITY DISTRO "1.4">
+<!ENTITY DISTRO_NAME "tbd">
+<!ENTITY YOCTO_DOC_VERSION "1.4">
+<!ENTITY POKYVERSION "8.0">
+<!ENTITY YOCTO_POKY "poky-&DISTRO_NAME;-&POKYVERSION;">
+<!ENTITY COPYRIGHT_YEAR "2010-2013">
+<!ENTITY YOCTO_DL_URL "http://downloads.yoctoproject.org">
+<!ENTITY YOCTO_HOME_URL "http://www.yoctoproject.org">
+<!ENTITY YOCTO_LISTS_URL "http://lists.yoctoproject.org">
+<!ENTITY YOCTO_BUGZILLA_URL "http://bugzilla.yoctoproject.org">
+<!ENTITY YOCTO_WIKI_URL "https://wiki.yoctoproject.org">
+<!ENTITY YOCTO_AB_URL "http://autobuilder.yoctoproject.org">
+<!ENTITY YOCTO_GIT_URL "http://git.yoctoproject.org">
+<!ENTITY YOCTO_ADTREPO_URL "http://adtrepo.yoctoproject.org">
+<!ENTITY OE_HOME_URL "http://www.openembedded.org">
+<!ENTITY OE_LISTS_URL "http://lists.linuxtogo.org/cgi-bin/mailman">
+<!ENTITY OE_DOCS_URL "http://docs.openembedded.org">
+<!ENTITY OH_HOME_URL "http://o-hand.com">
+<!ENTITY BITBAKE_HOME_URL "http://developer.berlios.de/projects/bitbake/">
+<!ENTITY ECLIPSE_MAIN_URL "http://www.eclipse.org/downloads">
+<!ENTITY ECLIPSE_DL_URL "http://download.eclipse.org">
+<!ENTITY ECLIPSE_DL_PLUGIN_URL "&YOCTO_DL_URL;/releases/eclipse-plugin/&DISTRO;">
+<!ENTITY ECLIPSE_UPDATES_URL "&ECLIPSE_DL_URL;/tm/updates/3.3">
+<!ENTITY ECLIPSE_INDIGO_URL "&ECLIPSE_DL_URL;/releases/indigo">
+<!ENTITY ECLIPSE_JUNO_URL "&ECLIPSE_DL_URL;/releases/juno">
+<!ENTITY ECLIPSE_INDIGO_CDT_URL "&ECLIPSE_DL_URL;tools/cdt/releases/indigo">
+<!ENTITY YOCTO_DOCS_URL "&YOCTO_HOME_URL;/docs">
+<!ENTITY YOCTO_SOURCES_URL "&YOCTO_HOME_URL;/sources/">
+<!ENTITY YOCTO_AB_PORT_URL "&YOCTO_AB_URL;:8010">
+<!ENTITY YOCTO_AB_NIGHTLY_URL "&YOCTO_AB_URL;/nightly/">
+<!ENTITY YOCTO_POKY_URL "&YOCTO_DL_URL;/releases/poky/">
+<!ENTITY YOCTO_RELEASE_DL_URL "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;">
+<!ENTITY YOCTO_TOOLCHAIN_DL_URL "&YOCTO_RELEASE_DL_URL;/toolchain/">
+<!ENTITY YOCTO_ECLIPSE_DL_URL "&YOCTO_RELEASE_DL_URL;/eclipse-plugin/indigo;">
+<!ENTITY YOCTO_ADTINSTALLER_DL_URL "&YOCTO_RELEASE_DL_URL;/adt_installer">
+<!ENTITY YOCTO_POKY_DL_URL "&YOCTO_RELEASE_DL_URL;/&YOCTO_POKY;.tar.bz2">
+<!ENTITY YOCTO_MACHINES_DL_URL "&YOCTO_RELEASE_DL_URL;/machines">
+<!ENTITY YOCTO_QEMU_DL_URL "&YOCTO_MACHINES_DL_URL;/qemu">
+<!ENTITY YOCTO_PYTHON-i686_DL_URL "&YOCTO_DL_URL;/releases/miscsupport/python-nativesdk-standalone-i686.tar.bz2">
+<!ENTITY YOCTO_PYTHON-x86_64_DL_URL "&YOCTO_DL_URL;/releases/miscsupport/python-nativesdk-standalone-x86_64.tar.bz2">
+<!ENTITY YOCTO_DOCS_QS_URL "&YOCTO_DOCS_URL;/&YOCTO_DOC_VERSION;/yocto-project-qs/yocto-project-qs.html">
+<!ENTITY YOCTO_DOCS_ADT_URL "&YOCTO_DOCS_URL;/&YOCTO_DOC_VERSION;/adt-manual/adt-manual.html">
+<!ENTITY YOCTO_DOCS_REF_URL "&YOCTO_DOCS_URL;/&YOCTO_DOC_VERSION;/ref-manual/ref-manual.html">
+<!ENTITY YOCTO_DOCS_BSP_URL "&YOCTO_DOCS_URL;/&YOCTO_DOC_VERSION;/bsp-guide/bsp-guide.html">
+<!ENTITY YOCTO_DOCS_DEV_URL "&YOCTO_DOCS_URL;/&YOCTO_DOC_VERSION;/dev-manual/dev-manual.html">
+<!ENTITY YOCTO_DOCS_KERNEL_URL "&YOCTO_DOCS_URL;/&YOCTO_DOC_VERSION;/kernel-manual/kernel-manual.html">
+<!ENTITY YOCTO_ADTPATH_DIR "/opt/poky/&DISTRO;">
+<!ENTITY YOCTO_POKY_TARBALL "&YOCTO_POKY;.tar.bz2">
+<!ENTITY OE_INIT_PATH "&YOCTO_POKY;/oe-init-build-env">
+<!ENTITY OE_INIT_FILE "oe-init-build-env">
+<!ENTITY UBUNTU_HOST_PACKAGES_ESSENTIAL "gawk wget git-core diffstat unzip texinfo \
+     build-essential chrpath">
+<!ENTITY FEDORA_HOST_PACKAGES_ESSENTIAL "gawk make wget tar bzip2 gzip python unzip perl patch \
+     diffutils diffstat git cpp gcc gcc-c++ eglibc-devel texinfo chrpath \
+     ccache">
+<!ENTITY OPENSUSE_HOST_PACKAGES_ESSENTIAL "python gcc gcc-c++ git chrpath make wget python-xml \
+     diffstat texinfo python-curses">
+<!ENTITY CENTOS_HOST_PACKAGES_ESSENTIAL "gawk make wget tar bzip2 gzip python unzip perl patch \
+     diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath">

bitbake/doc/template/component.title.xsl

@@ -0,0 +1,39 @@
+<xsl:stylesheet version="1.0"
+  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
+  xmlns:d="http://docbook.org/ns/docbook"
+  xmlns="http://www.w3.org/1999/xhtml"
+  exclude-result-prefixes="d">
+  
+  <xsl:template name="component.title">
+    <xsl:param name="node" select="."/>
+    
+    <xsl:variable name="level">
+      <xsl:choose>
+        <xsl:when test="ancestor::d:section">
+          <xsl:value-of select="count(ancestor::d:section)+1"/>
+        </xsl:when>
+        <xsl:when test="ancestor::d:sect5">6</xsl:when>
+        <xsl:when test="ancestor::d:sect4">5</xsl:when>
+        <xsl:when test="ancestor::d:sect3">4</xsl:when>
+        <xsl:when test="ancestor::d:sect2">3</xsl:when>
+        <xsl:when test="ancestor::d:sect1">2</xsl:when>
+        <xsl:otherwise>1</xsl:otherwise>
+      </xsl:choose>
+    </xsl:variable>
+    <xsl:element name="h{$level+1}" namespace="http://www.w3.org/1999/xhtml">
+      <xsl:attribute name="class">title</xsl:attribute>
+      <xsl:if test="$generate.id.attributes = 0">
+        <xsl:call-template name="anchor">
+          <xsl:with-param name="node" select="$node"/>
+          <xsl:with-param name="conditional" select="0"/>
+        </xsl:call-template>
+      </xsl:if>
+      <xsl:apply-templates select="$node" mode="object.title.markup">
+        <xsl:with-param name="allow-anchors" select="1"/>
+      </xsl:apply-templates>
+      <xsl:call-template name="permalink">
+        <xsl:with-param name="node" select="$node"/>
+      </xsl:call-template>
+    </xsl:element>
+  </xsl:template>
+</xsl:stylesheet>

bitbake/doc/template/db-pdf.xsl

@@ -0,0 +1,64 @@
+<?xml version='1.0'?>
+<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://www.w3.org/1999/xhtml" xmlns:fo="http://www.w3.org/1999/XSL/Format" version="1.0">
+  
+  <xsl:import href="http://docbook.sourceforge.net/release/xsl/current/fo/docbook.xsl" />
+
+  <!-- check project-plan.sh for how this is generated, needed to tweak 
+       the cover page     
+    -->
+  <xsl:include href="/tmp/titlepage.xsl"/> 
+
+  <!-- To force a page break in document, i.e per section add a 
+      <?hard-pagebreak?> tag.
+  -->
+ <xsl:template match="processing-instruction('hard-pagebreak')">
+   <fo:block break-before='page' />
+ </xsl:template>
+
+  <!--Fix for defualt indent getting TOC all wierd..
+      See http://sources.redhat.com/ml/docbook-apps/2005-q1/msg00455.html 
+      FIXME: must be a better fix
+    -->
+  <xsl:param name="body.start.indent" select="'0'"/>
+  <!--<xsl:param name="title.margin.left" select="'0'"/>-->
+
+  <!-- stop long-ish header titles getting wrapped -->
+  <xsl:param name="header.column.widths">1 10 1</xsl:param>
+
+  <!-- customise headers and footers a little --> 
+
+  <xsl:template name="head.sep.rule">
+   <xsl:if test="$header.rule != 0">
+     <xsl:attribute name="border-bottom-width">0.5pt</xsl:attribute>
+     <xsl:attribute name="border-bottom-style">solid</xsl:attribute>
+     <xsl:attribute name="border-bottom-color">#cccccc</xsl:attribute>
+   </xsl:if>
+  </xsl:template>
+
+  <xsl:template name="foot.sep.rule">
+    <xsl:if test="$footer.rule != 0">
+     <xsl:attribute name="border-top-width">0.5pt</xsl:attribute>
+     <xsl:attribute name="border-top-style">solid</xsl:attribute>
+     <xsl:attribute name="border-top-color">#cccccc</xsl:attribute>
+    </xsl:if>
+  </xsl:template>
+
+  <xsl:attribute-set name="header.content.properties">
+    <xsl:attribute name="color">#cccccc</xsl:attribute>
+  </xsl:attribute-set>
+
+  <xsl:attribute-set name="footer.content.properties">
+    <xsl:attribute name="color">#cccccc</xsl:attribute>
+  </xsl:attribute-set>
+
+ 
+  <!-- general settings -->
+
+  <xsl:param name="fop1.extensions" select="1"></xsl:param>
+  <xsl:param name="paper.type" select="'A4'"></xsl:param>
+  <xsl:param name="section.autolabel" select="1"></xsl:param>
+  <xsl:param name="body.font.family" select="'verasans'"></xsl:param>
+  <xsl:param name="title.font.family" select="'verasans'"></xsl:param>
+  <xsl:param name="monospace.font.family" select="'veramono'"></xsl:param>
+
+</xsl:stylesheet>

bitbake/doc/template/division.title.xsl

@@ -0,0 +1,25 @@
+<xsl:stylesheet version="1.0"
+  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
+  xmlns:d="http://docbook.org/ns/docbook"
+  xmlns="http://www.w3.org/1999/xhtml"
+  exclude-result-prefixes="d">
+  
+  <xsl:template name="division.title">
+    <xsl:param name="node" select="."/>
+    
+    <h1>
+      <xsl:attribute name="class">title</xsl:attribute>
+      <xsl:call-template name="anchor">
+        <xsl:with-param name="node" select="$node"/>
+        <xsl:with-param name="conditional" select="0"/>
+      </xsl:call-template>
+      <xsl:apply-templates select="$node" mode="object.title.markup">
+        <xsl:with-param name="allow-anchors" select="1"/>
+      </xsl:apply-templates>
+      <xsl:call-template name="permalink">
+        <xsl:with-param name="node" select="$node"/>
+      </xsl:call-template>
+    </h1>
+  </xsl:template>
+</xsl:stylesheet>
+

bitbake/doc/template/fop-config.xml

@@ -0,0 +1,58 @@
+<fop version="1.0">
+
+  <!-- Strict user configuration -->
+  <strict-configuration>true</strict-configuration>
+
+  <!-- Strict FO validation -->
+  <strict-validation>true</strict-validation>
+
+   <!--
+    Set the baseDir so common/openedhand.svg references in plans still
+    work ok. Note, relative file references to current dir should still work.
+    -->	
+  <base>../template</base>
+  <font-base>../template</font-base>
+ 
+  <!-- Source resolution in dpi (dots/pixels per inch) for determining the
+       size of pixels in SVG and bitmap images, default: 72dpi -->
+  <!-- <source-resolution>72</source-resolution> -->
+  <!-- Target resolution in dpi (dots/pixels per inch) for specifying the
+       target resolution for generated bitmaps, default: 72dpi -->
+  <!-- <target-resolution>72</target-resolution> -->
+ 
+  <!-- default page-height and page-width, in case
+       value is specified as auto -->
+  <default-page-settings height="11in" width="8.26in"/> 
+ 
+  <!-- <use-cache>false</use-cache> -->
+ 
+  <renderers>
+    <renderer mime="application/pdf">
+      <fonts>
+        <font  metrics-file="VeraMono.xml"
+               kerning="yes" 
+               embed-url="VeraMono.ttf">
+          <font-triplet name="veramono" style="normal" weight="normal"/>
+        </font>
+
+        <font  metrics-file="VeraMoBd.xml"
+               kerning="yes" 
+               embed-url="VeraMoBd.ttf">
+          <font-triplet name="veramono" style="normal" weight="bold"/>
+        </font>
+
+        <font  metrics-file="Vera.xml"
+               kerning="yes" 
+               embed-url="Vera.ttf">
+          <font-triplet name="verasans" style="normal" weight="normal"/>
+          <font-triplet name="verasans" style="normal" weight="bold"/>
+          <font-triplet name="verasans" style="italic" weight="normal"/>
+          <font-triplet name="verasans" style="italic" weight="bold"/>
+        </font>
+        
+        <auto-detect/>
+      </fonts>
+    </renderer>
+  </renderers>
+</fop>
+

bitbake/doc/template/formal.object.heading.xsl

@@ -0,0 +1,21 @@
+<xsl:stylesheet version="1.0"
+  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
+  xmlns:d="http://docbook.org/ns/docbook"
+  xmlns="http://www.w3.org/1999/xhtml"
+  exclude-result-prefixes="d">
+  
+  <xsl:template name="formal.object.heading">
+    <xsl:param name="object" select="."/>
+    <xsl:param name="title">
+      <xsl:apply-templates select="$object" mode="object.title.markup">
+        <xsl:with-param name="allow-anchors" select="1"/>
+      </xsl:apply-templates>
+    </xsl:param>
+    <p class="title">
+      <b><xsl:copy-of select="$title"/></b>
+      <xsl:call-template name="permalink">
+        <xsl:with-param name="node" select="$object"/>
+      </xsl:call-template>
+    </p>
+  </xsl:template>
+</xsl:stylesheet>

bitbake/doc/template/gloss-permalinks.xsl

@@ -0,0 +1,14 @@
+<xsl:stylesheet version="1.0"
+  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
+  xmlns:d="http://docbook.org/ns/docbook"
+  xmlns="http://www.w3.org/1999/xhtml">
+
+  <xsl:template match="glossentry/glossterm">
+    <xsl:apply-imports/>
+    <xsl:if test="$generate.permalink != 0">
+      <xsl:call-template name="permalink">
+        <xsl:with-param name="node" select=".."/>
+      </xsl:call-template>
+    </xsl:if>
+  </xsl:template>
+</xsl:stylesheet>

bitbake/doc/template/permalinks.xsl

@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<xsl:stylesheet version="1.0"
+  xmlns="http://www.w3.org/1999/xhtml"
+  xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
+
+  <xsl:param name="generate.permalink" select="1"/>
+  <xsl:param name="permalink.text">¶</xsl:param>
+
+  <xsl:template name="permalink">
+    <xsl:param name="node"/>
+
+    <xsl:if test="$generate.permalink != '0'">
+      <span class="permalink">
+        <a alt="Permalink" title="Permalink">
+          <xsl:attribute name="href">
+            <xsl:call-template name="href.target">
+              <xsl:with-param name="object"  select="$node"/>
+            </xsl:call-template>
+          </xsl:attribute>
+          <xsl:copy-of select="$permalink.text"/>
+        </a>
+      </span>
+    </xsl:if>
+  </xsl:template>
+</xsl:stylesheet>

bitbake/doc/template/section.title.xsl

@@ -0,0 +1,55 @@
+<xsl:stylesheet version="1.0"
+  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
+  xmlns:d="http://docbook.org/ns/docbook"
+  xmlns="http://www.w3.org/1999/xhtml" exclude-result-prefixes="d">
+
+  <xsl:template name="section.title">
+    <xsl:variable name="section"
+      select="(ancestor::section |
+               ancestor::simplesect|
+               ancestor::sect1|
+               ancestor::sect2|
+               ancestor::sect3|
+               ancestor::sect4|
+               ancestor::sect5)[last()]"/>
+
+    <xsl:variable name="renderas">
+      <xsl:choose>
+        <xsl:when test="$section/@renderas = 'sect1'">1</xsl:when>
+        <xsl:when test="$section/@renderas = 'sect2'">2</xsl:when>
+        <xsl:when test="$section/@renderas = 'sect3'">3</xsl:when>
+        <xsl:when test="$section/@renderas = 'sect4'">4</xsl:when>
+        <xsl:when test="$section/@renderas = 'sect5'">5</xsl:when>
+        <xsl:otherwise><xsl:value-of select="''"/></xsl:otherwise>
+      </xsl:choose>
+    </xsl:variable>
+
+    <xsl:variable name="level">
+      <xsl:choose>
+        <xsl:when test="$renderas != ''">
+          <xsl:value-of select="$renderas"/>
+        </xsl:when>
+        <xsl:otherwise>
+          <xsl:call-template name="section.level">
+            <xsl:with-param name="node" select="$section"/>
+          </xsl:call-template>
+        </xsl:otherwise>
+      </xsl:choose>
+    </xsl:variable>
+
+    <xsl:call-template name="section.heading">
+      <xsl:with-param name="section" select="$section"/>
+      <xsl:with-param name="level" select="$level"/>
+      <xsl:with-param name="title">
+        <xsl:apply-templates select="$section" mode="object.title.markup">
+          <xsl:with-param name="allow-anchors" select="1"/>
+        </xsl:apply-templates>
+        <xsl:if test="$level &gt; 0">
+          <xsl:call-template name="permalink">
+            <xsl:with-param name="node" select="$section"/>
+          </xsl:call-template>
+        </xsl:if>
+      </xsl:with-param>
+    </xsl:call-template>
+  </xsl:template>
+</xsl:stylesheet>

bitbake/doc/tools/docbook-to-pdf

@@ -0,0 +1,51 @@
+#!/bin/sh
+	
+if [ -z "$1" -o -z "$2" ]; then
+   echo "usage: [-v] $0 <docbook file> <templatedir>"
+   echo
+   echo "*NOTE* you need xsltproc, fop and nwalsh docbook stylesheets" 
+   echo "       installed for this to work!"
+   echo
+   exit 0
+fi
+
+FO=`echo $1 | sed s/.xml/.fo/` || exit 1
+PDF=`echo $1 | sed s/.xml/.pdf/` || exit 1
+TEMPLATEDIR=$2
+
+##
+# These URI should be rewritten by your distribution's xml catalog to
+# match your localy installed XSL stylesheets.
+XSL_BASE_URI="http://docbook.sourceforge.net/release/xsl/current"
+
+# Creates a temporary XSL stylesheet based on titlepage.xsl
+xsltproc -o /tmp/titlepage.xsl                                           \
+	 --xinclude                                                      \
+         $XSL_BASE_URI/template/titlepage.xsl \
+         $TEMPLATEDIR/titlepage.templates.xml || exit 1
+
+# Creates the file needed for FOP
+xsltproc --xinclude                    \
+	 --stringparam hyphenate false \
+	 --stringparam formal.title.placement "figure after" \
+	 --stringparam ulink.show 1 \
+         --stringparam  body.font.master  9 \
+         --stringparam  title.font.master  11 \
+         --stringparam draft.watermark.image "$TEMPLATEDIR/draft.png" \
+         --stringparam  chapter.autolabel 1 \
+         --stringparam  appendix.autolabel A \
+         --stringparam  section.autolabel 1 \
+         --stringparam  section.label.includes.component.label 1 \
+         --output $FO               \
+         $TEMPLATEDIR/db-pdf.xsl    \
+	 $1                 || exit 1
+
+# Invokes the Java version of FOP.  Uses the additional configuration file common/fop-config.xml
+fop -c $TEMPLATEDIR/fop-config.xml -fo $FO -pdf $PDF       || exit 1
+
+rm -f $FO
+rm -f  /tmp/titlepage.xsl
+
+echo
+echo " #### Success! $PDF ready. ####"
+echo

bitbake/lib/bb/COW.py

@@ -0,0 +1,323 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# This is a copy on write dictionary and set which abuses classes to try and be nice and fast.
+#
+# Copyright (C) 2006 Tim Amsell
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+#Please Note:
+# Be careful when using mutable types (ie Dict and Lists) - operations involving these are SLOW.
+# Assign a file to __warn__ to get warnings about slow operations.
+#
+
+from __future__ import print_function
+import copy
+import types
+ImmutableTypes = (
+    types.NoneType,
+    bool,
+    complex,
+    float,
+    int,
+    long,
+    tuple,
+    frozenset,
+    basestring
+)
+
+MUTABLE = "__mutable__"
+
+class COWMeta(type):
+    pass
+
+class COWDictMeta(COWMeta):
+    __warn__ = False
+    __hasmutable__ = False
+    __marker__ = tuple()
+
+    def __str__(cls):
+        # FIXME: I have magic numbers!
+        return "<COWDict Level: %i Current Keys: %i>" % (cls.__count__, len(cls.__dict__) - 3)
+    __repr__ = __str__
+
+    def cow(cls):
+        class C(cls):
+            __count__ = cls.__count__ + 1
+        return C
+    copy = cow
+    __call__ = cow
+
+    def __setitem__(cls, key, value):
+        if not isinstance(value, ImmutableTypes):
+            if not isinstance(value, COWMeta):
+                cls.__hasmutable__ = True
+            key += MUTABLE
+        setattr(cls, key, value)
+
+    def __getmutable__(cls, key, readonly=False):
+        nkey = key + MUTABLE
+        try:
+            return cls.__dict__[nkey]
+        except KeyError:
+            pass
+
+        value = getattr(cls, nkey)
+        if readonly:
+            return value
+
+        if not cls.__warn__ is False and not isinstance(value, COWMeta):
+            print("Warning: Doing a copy because %s is a mutable type." % key, file=cls.__warn__)
+        try:
+            value = value.copy()
+        except AttributeError as e:
+            value = copy.copy(value)
+        setattr(cls, nkey, value)
+        return value
+
+    __getmarker__ = []
+    def __getreadonly__(cls, key, default=__getmarker__):
+        """\
+        Get a value (even if mutable) which you promise not to change.
+        """
+        return cls.__getitem__(key, default, True)
+
+    def __getitem__(cls, key, default=__getmarker__, readonly=False):
+        try:
+            try:
+                value = getattr(cls, key)
+            except AttributeError:
+                value = cls.__getmutable__(key, readonly)
+
+            # This is for values which have been deleted
+            if value is cls.__marker__:
+                raise AttributeError("key %s does not exist." % key)
+
+            return value
+        except AttributeError as e:
+            if not default is cls.__getmarker__:
+                return default
+
+            raise KeyError(str(e))
+
+    def __delitem__(cls, key):
+        cls.__setitem__(key, cls.__marker__)
+
+    def __revertitem__(cls, key):
+        if not cls.__dict__.has_key(key):
+            key += MUTABLE
+        delattr(cls, key)
+
+    def __contains__(cls, key):
+        return cls.has_key(key)
+
+    def has_key(cls, key):
+        value = cls.__getreadonly__(key, cls.__marker__)
+        if value is cls.__marker__:
+            return False
+        return True
+
+    def iter(cls, type, readonly=False):
+        for key in dir(cls):
+            if key.startswith("__"):
+                continue
+
+            if key.endswith(MUTABLE):
+                key = key[:-len(MUTABLE)]
+
+            if type == "keys":
+                yield key
+
+            try:
+                if readonly:
+                    value = cls.__getreadonly__(key)
+                else:
+                    value = cls[key]
+            except KeyError:
+                continue
+
+            if type == "values":
+                yield value
+            if type == "items":
+                yield (key, value)
+        raise StopIteration()
+
+    def iterkeys(cls):
+        return cls.iter("keys")
+    def itervalues(cls, readonly=False):
+        if not cls.__warn__ is False and cls.__hasmutable__ and readonly is False:
+            print("Warning: If you arn't going to change any of the values call with True.", file=cls.__warn__)
+        return cls.iter("values", readonly)
+    def iteritems(cls, readonly=False):
+        if not cls.__warn__ is False and cls.__hasmutable__ and readonly is False:
+            print("Warning: If you arn't going to change any of the values call with True.", file=cls.__warn__)
+        return cls.iter("items", readonly)
+
+class COWSetMeta(COWDictMeta):
+    def __str__(cls):
+        # FIXME: I have magic numbers!
+        return "<COWSet Level: %i Current Keys: %i>" % (cls.__count__, len(cls.__dict__) -3)
+    __repr__ = __str__
+
+    def cow(cls):
+        class C(cls):
+            __count__ = cls.__count__ + 1
+        return C
+
+    def add(cls, value):
+        COWDictMeta.__setitem__(cls, repr(hash(value)), value)
+
+    def remove(cls, value):
+        COWDictMeta.__delitem__(cls, repr(hash(value)))
+
+    def __in__(cls, value):
+        return COWDictMeta.has_key(repr(hash(value)))
+
+    def iterkeys(cls):
+        raise TypeError("sets don't have keys")
+
+    def iteritems(cls):
+        raise TypeError("sets don't have 'items'")
+
+# These are the actual classes you use!
+class COWDictBase(object):
+    __metaclass__ = COWDictMeta
+    __count__ = 0
+
+class COWSetBase(object):
+    __metaclass__ = COWSetMeta
+    __count__ = 0
+
+if __name__ == "__main__":
+    import sys
+    COWDictBase.__warn__ = sys.stderr
+    a = COWDictBase()
+    print("a", a)
+
+    a['a'] = 'a'
+    a['b'] = 'b'
+    a['dict'] = {}
+
+    b = a.copy()
+    print("b", b)
+    b['c'] = 'b'
+
+    print()
+
+    print("a", a)
+    for x in a.iteritems():
+        print(x)
+    print("--")
+    print("b", b)
+    for x in b.iteritems():
+        print(x)
+    print()
+
+    b['dict']['a'] = 'b'
+    b['a'] = 'c'
+
+    print("a", a)
+    for x in a.iteritems():
+        print(x)
+    print("--")
+    print("b", b)
+    for x in b.iteritems():
+        print(x)
+    print()
+
+    try:
+        b['dict2']
+    except KeyError as e:
+        print("Okay!")
+
+    a['set'] = COWSetBase()
+    a['set'].add("o1")
+    a['set'].add("o1")
+    a['set'].add("o2")
+
+    print("a", a)
+    for x in a['set'].itervalues():
+        print(x)
+    print("--")
+    print("b", b)
+    for x in b['set'].itervalues():
+        print(x)
+    print()
+
+    b['set'].add('o3')
+
+    print("a", a)
+    for x in a['set'].itervalues():
+        print(x)
+    print("--")
+    print("b", b)
+    for x in b['set'].itervalues():
+        print(x)
+    print()
+
+    a['set2'] = set()
+    a['set2'].add("o1")
+    a['set2'].add("o1")
+    a['set2'].add("o2")
+
+    print("a", a)
+    for x in a.iteritems():
+        print(x)
+    print("--")
+    print("b", b)
+    for x in b.iteritems(readonly=True):
+        print(x)
+    print()
+
+    del b['b']
+    try:
+        print(b['b'])
+    except KeyError:
+        print("Yay! deleted key raises error")
+
+    if b.has_key('b'):
+        print("Boo!")
+    else:
+        print("Yay - has_key with delete works!")
+
+    print("a", a)
+    for x in a.iteritems():
+        print(x)
+    print("--")
+    print("b", b)
+    for x in b.iteritems(readonly=True):
+        print(x)
+    print()
+
+    b.__revertitem__('b')
+
+    print("a", a)
+    for x in a.iteritems():
+        print(x)
+    print("--")
+    print("b", b)
+    for x in b.iteritems(readonly=True):
+        print(x)
+    print()
+
+    b.__revertitem__('dict')
+    print("a", a)
+    for x in a.iteritems():
+        print(x)
+    print("--")
+    print("b", b)
+    for x in b.iteritems(readonly=True):
+        print(x)
+    print()

bitbake/lib/bb/__init__.py

@@ -0,0 +1,144 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# BitBake Build System Python Library
+#
+# Copyright (C) 2003  Holger Schurig
+# Copyright (C) 2003, 2004  Chris Larson
+#
+# Based on Gentoo's portage.py.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+__version__ = "1.30.0"
+
+import sys
+if sys.version_info < (2, 7, 3):
+    raise RuntimeError("Sorry, python 2.7.3 or later is required for this version of bitbake")
+
+
+class BBHandledException(Exception):
+    """
+    The big dilemma for generic bitbake code is what information to give the user
+    when an exception occurs. Any exception inheriting this base exception class
+    has already provided information to the user via some 'fired' message type such as
+    an explicitly fired event using bb.fire, or a bb.error message. If bitbake 
+    encounters an exception derived from this class, no backtrace or other information 
+    will be given to the user, its assumed the earlier event provided the relevant information.
+    """
+    pass
+
+import os
+import logging
+
+
+class NullHandler(logging.Handler):
+    def emit(self, record):
+        pass
+
+Logger = logging.getLoggerClass()
+class BBLogger(Logger):
+    def __init__(self, name):
+        if name.split(".")[0] == "BitBake":
+            self.debug = self.bbdebug
+        Logger.__init__(self, name)
+
+    def bbdebug(self, level, msg, *args, **kwargs):
+        return self.log(logging.DEBUG - level + 1, msg, *args, **kwargs)
+
+    def plain(self, msg, *args, **kwargs):
+        return self.log(logging.INFO + 1, msg, *args, **kwargs)
+
+    def verbose(self, msg, *args, **kwargs):
+        return self.log(logging.INFO - 1, msg, *args, **kwargs)
+
+logging.raiseExceptions = False
+logging.setLoggerClass(BBLogger)
+
+logger = logging.getLogger("BitBake")
+logger.addHandler(NullHandler())
+logger.setLevel(logging.DEBUG - 2)
+
+mainlogger = logging.getLogger("BitBake.Main")
+
+# This has to be imported after the setLoggerClass, as the import of bb.msg
+# can result in construction of the various loggers.
+import bb.msg
+
+from bb import fetch2 as fetch
+sys.modules['bb.fetch'] = sys.modules['bb.fetch2']
+
+# Messaging convenience functions
+def plain(*args):
+    mainlogger.plain(''.join(args))
+
+def debug(lvl, *args):
+    if isinstance(lvl, basestring):
+        mainlogger.warn("Passed invalid debug level '%s' to bb.debug", lvl)
+        args = (lvl,) + args
+        lvl = 1
+    mainlogger.debug(lvl, ''.join(args))
+
+def note(*args):
+    mainlogger.info(''.join(args))
+
+def warn(*args):
+    mainlogger.warn(''.join(args))
+
+def error(*args, **kwargs):
+    mainlogger.error(''.join(args), extra=kwargs)
+
+def fatal(*args, **kwargs):
+    mainlogger.critical(''.join(args), extra=kwargs)
+    raise BBHandledException()
+
+def deprecated(func, name=None, advice=""):
+    """This is a decorator which can be used to mark functions
+    as deprecated. It will result in a warning being emitted
+    when the function is used."""
+    import warnings
+
+    if advice:
+        advice = ": %s" % advice
+    if name is None:
+        name = func.__name__
+
+    def newFunc(*args, **kwargs):
+        warnings.warn("Call to deprecated function %s%s." % (name,
+                                                             advice),
+                      category=DeprecationWarning,
+                      stacklevel=2)
+        return func(*args, **kwargs)
+    newFunc.__name__ = func.__name__
+    newFunc.__doc__ = func.__doc__
+    newFunc.__dict__.update(func.__dict__)
+    return newFunc
+
+# For compatibility
+def deprecate_import(current, modulename, fromlist, renames = None):
+    """Import objects from one module into another, wrapping them with a DeprecationWarning"""
+    import sys
+
+    module = __import__(modulename, fromlist = fromlist)
+    for position, objname in enumerate(fromlist):
+        obj = getattr(module, objname)
+        newobj = deprecated(obj, "{0}.{1}".format(current, objname),
+                            "Please use {0}.{1} instead".format(modulename, objname))
+        if renames:
+            newname = renames[position]
+        else:
+            newname = objname
+
+        setattr(sys.modules[current], newname, newobj)
+

bitbake/lib/bb/build.py

@@ -0,0 +1,784 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# BitBake 'Build' implementation
+#
+# Core code for function execution and task handling in the
+# BitBake build tools.
+#
+# Copyright (C) 2003, 2004  Chris Larson
+#
+# Based on Gentoo's portage.py.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+import os
+import sys
+import logging
+import shlex
+import glob
+import time
+import stat
+import bb
+import bb.msg
+import bb.process
+from contextlib import nested
+from bb import event, utils
+
+bblogger = logging.getLogger('BitBake')
+logger = logging.getLogger('BitBake.Build')
+
+NULL = open(os.devnull, 'r+')
+
+__mtime_cache = {}
+
+def cached_mtime_noerror(f):
+    if f not in __mtime_cache:
+        try:
+            __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
+        except OSError:
+            return 0
+    return __mtime_cache[f]
+
+def reset_cache():
+    global __mtime_cache
+    __mtime_cache = {}
+
+# When we execute a Python function, we'd like certain things
+# in all namespaces, hence we add them to __builtins__.
+# If we do not do this and use the exec globals, they will
+# not be available to subfunctions.
+__builtins__['bb'] = bb
+__builtins__['os'] = os
+
+class FuncFailed(Exception):
+    def __init__(self, name = None, logfile = None):
+        self.logfile = logfile
+        self.name = name
+        if name:
+            self.msg = 'Function failed: %s' % name
+        else:
+            self.msg = "Function failed"
+
+    def __str__(self):
+        if self.logfile and os.path.exists(self.logfile):
+            msg = ("%s (log file is located at %s)" %
+                   (self.msg, self.logfile))
+        else:
+            msg = self.msg
+        return msg
+
+class TaskBase(event.Event):
+    """Base class for task events"""
+
+    def __init__(self, t, logfile, d):
+        self._task = t
+        self._package = d.getVar("PF", True)
+        self.taskfile = d.getVar("FILE", True)
+        self.taskname = self._task
+        self.logfile = logfile
+        self.time = time.time()
+        event.Event.__init__(self)
+        self._message = "recipe %s: task %s: %s" % (d.getVar("PF", True), t, self.getDisplayName())
+
+    def getTask(self):
+        return self._task
+
+    def setTask(self, task):
+        self._task = task
+
+    def getDisplayName(self):
+        return bb.event.getName(self)[4:]
+
+    task = property(getTask, setTask, None, "task property")
+
+class TaskStarted(TaskBase):
+    """Task execution started"""
+    def __init__(self, t, logfile, taskflags, d):
+        super(TaskStarted, self).__init__(t, logfile, d)
+        self.taskflags = taskflags
+
+class TaskSucceeded(TaskBase):
+    """Task execution completed"""
+
+class TaskFailed(TaskBase):
+    """Task execution failed"""
+
+    def __init__(self, task, logfile, metadata, errprinted = False):
+        self.errprinted = errprinted
+        super(TaskFailed, self).__init__(task, logfile, metadata)
+
+class TaskFailedSilent(TaskBase):
+    """Task execution failed (silently)"""
+    def getDisplayName(self):
+        # Don't need to tell the user it was silent
+        return "Failed"
+
+class TaskInvalid(TaskBase):
+
+    def __init__(self, task, metadata):
+        super(TaskInvalid, self).__init__(task, None, metadata)
+        self._message = "No such task '%s'" % task
+
+
+class LogTee(object):
+    def __init__(self, logger, outfile):
+        self.outfile = outfile
+        self.logger = logger
+        self.name = self.outfile.name
+
+    def write(self, string):
+        self.logger.plain(string)
+        self.outfile.write(string)
+
+    def __enter__(self):
+        self.outfile.__enter__()
+        return self
+
+    def __exit__(self, *excinfo):
+        self.outfile.__exit__(*excinfo)
+
+    def __repr__(self):
+        return '<LogTee {0}>'.format(self.name)
+    def flush(self):
+        self.outfile.flush()
+
+#
+# pythonexception allows the python exceptions generated to be raised
+# as the real exceptions (not FuncFailed) and without a backtrace at the 
+# origin of the failure.
+#
+def exec_func(func, d, dirs = None, pythonexception=False):
+    """Execute a BB 'function'"""
+
+    body = d.getVar(func, False)
+    if not body:
+        if body is None:
+            logger.warn("Function %s doesn't exist", func)
+        return
+
+    flags = d.getVarFlags(func)
+    cleandirs = flags.get('cleandirs')
+    if cleandirs:
+        for cdir in d.expand(cleandirs).split():
+            bb.utils.remove(cdir, True)
+            bb.utils.mkdirhier(cdir)
+
+    if dirs is None:
+        dirs = flags.get('dirs')
+        if dirs:
+            dirs = d.expand(dirs).split()
+
+    if dirs:
+        for adir in dirs:
+            bb.utils.mkdirhier(adir)
+        adir = dirs[-1]
+    else:
+        adir = d.getVar('B', True)
+        bb.utils.mkdirhier(adir)
+
+    ispython = flags.get('python')
+
+    lockflag = flags.get('lockfiles')
+    if lockflag:
+        lockfiles = [f for f in d.expand(lockflag).split()]
+    else:
+        lockfiles = None
+
+    tempdir = d.getVar('T', True)
+
+    # or func allows items to be executed outside of the normal
+    # task set, such as buildhistory
+    task = d.getVar('BB_RUNTASK', True) or func
+    if task == func:
+        taskfunc = task
+    else:
+        taskfunc = "%s.%s" % (task, func)
+
+    runfmt = d.getVar('BB_RUNFMT', True) or "run.{func}.{pid}"
+    runfn = runfmt.format(taskfunc=taskfunc, task=task, func=func, pid=os.getpid())
+    runfile = os.path.join(tempdir, runfn)
+    bb.utils.mkdirhier(os.path.dirname(runfile))
+
+    # Setup the courtesy link to the runfn, only for tasks
+    # we create the link 'just' before the run script is created
+    # if we create it after, and if the run script fails, then the
+    # link won't be created as an exception would be fired.
+    if task == func:
+        runlink = os.path.join(tempdir, 'run.{0}'.format(task))
+        if runlink:
+            bb.utils.remove(runlink)
+
+            try:
+                os.symlink(runfn, runlink)
+            except OSError:
+                pass
+
+    with bb.utils.fileslocked(lockfiles):
+        if ispython:
+            exec_func_python(func, d, runfile, cwd=adir, pythonexception=pythonexception)
+        else:
+            exec_func_shell(func, d, runfile, cwd=adir)
+
+_functionfmt = """
+{function}(d)
+"""
+logformatter = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
+def exec_func_python(func, d, runfile, cwd=None, pythonexception=False):
+    """Execute a python BB 'function'"""
+
+    code = _functionfmt.format(function=func)
+    bb.utils.mkdirhier(os.path.dirname(runfile))
+    with open(runfile, 'w') as script:
+        bb.data.emit_func_python(func, script, d)
+
+    if cwd:
+        try:
+            olddir = os.getcwd()
+        except OSError:
+            olddir = None
+        os.chdir(cwd)
+
+    bb.debug(2, "Executing python function %s" % func)
+
+    try:
+        text = "def %s(d):\n%s" % (func, d.getVar(func, False))
+        fn = d.getVarFlag(func, "filename", False)
+        lineno = int(d.getVarFlag(func, "lineno", False))
+        bb.methodpool.insert_method(func, text, fn, lineno - 1)
+
+        comp = utils.better_compile(code, func, "exec_python_func() autogenerated")
+        utils.better_exec(comp, {"d": d}, code, "exec_python_func() autogenerated", pythonexception=pythonexception)
+    except (bb.parse.SkipRecipe, bb.build.FuncFailed):
+        raise
+    except:
+        if pythonexception:
+            raise
+        raise FuncFailed(func, None)
+    finally:
+        bb.debug(2, "Python function %s finished" % func)
+
+        if cwd and olddir:
+            try:
+                os.chdir(olddir)
+            except OSError:
+                pass
+
+def shell_trap_code():
+    return '''#!/bin/sh\n
+# Emit a useful diagnostic if something fails:
+bb_exit_handler() {
+    ret=$?
+    case $ret in
+    0)  ;;
+    *)  case $BASH_VERSION in
+        "") echo "WARNING: exit code $ret from a shell command.";;
+        *)  echo "WARNING: ${BASH_SOURCE[0]}:${BASH_LINENO[0]} exit $ret from '$BASH_COMMAND'";;
+        esac
+        exit $ret
+    esac
+}
+trap 'bb_exit_handler' 0
+set -e
+'''
+
+def exec_func_shell(func, d, runfile, cwd=None):
+    """Execute a shell function from the metadata
+
+    Note on directory behavior.  The 'dirs' varflag should contain a list
+    of the directories you need created prior to execution.  The last
+    item in the list is where we will chdir/cd to.
+    """
+
+    # Don't let the emitted shell script override PWD
+    d.delVarFlag('PWD', 'export')
+
+    with open(runfile, 'w') as script:
+        script.write(shell_trap_code())
+
+        bb.data.emit_func(func, script, d)
+
+        if bb.msg.loggerVerboseLogs:
+            script.write("set -x\n")
+        if cwd:
+            script.write("cd '%s'\n" % cwd)
+        script.write("%s\n" % func)
+        script.write('''
+# cleanup
+ret=$?
+trap '' 0
+exit $ret
+''')
+
+    os.chmod(runfile, 0775)
+
+    cmd = runfile
+    if d.getVarFlag(func, 'fakeroot', False):
+        fakerootcmd = d.getVar('FAKEROOT', True)
+        if fakerootcmd:
+            cmd = [fakerootcmd, runfile]
+
+    if bb.msg.loggerDefaultVerbose:
+        logfile = LogTee(logger, sys.stdout)
+    else:
+        logfile = sys.stdout
+
+    def readfifo(data):
+        lines = data.split('\0')
+        for line in lines:
+            splitval = line.split(' ', 1)
+            cmd = splitval[0]
+            if len(splitval) > 1:
+                value = splitval[1]
+            else:
+                value = ''
+            if cmd == 'bbplain':
+                bb.plain(value)
+            elif cmd == 'bbnote':
+                bb.note(value)
+            elif cmd == 'bbwarn':
+                bb.warn(value)
+            elif cmd == 'bberror':
+                bb.error(value)
+            elif cmd == 'bbfatal':
+                # The caller will call exit themselves, so bb.error() is
+                # what we want here rather than bb.fatal()
+                bb.error(value)
+            elif cmd == 'bbfatal_log':
+                bb.error(value, forcelog=True)
+            elif cmd == 'bbdebug':
+                splitval = value.split(' ', 1)
+                level = int(splitval[0])
+                value = splitval[1]
+                bb.debug(level, value)
+
+    tempdir = d.getVar('T', True)
+    fifopath = os.path.join(tempdir, 'fifo.%s' % os.getpid())
+    if os.path.exists(fifopath):
+        os.unlink(fifopath)
+    os.mkfifo(fifopath)
+    with open(fifopath, 'r+') as fifo:
+        try:
+            bb.debug(2, "Executing shell function %s" % func)
+
+            try:
+                with open(os.devnull, 'r+') as stdin:
+                    bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)])
+            except bb.process.CmdError:
+                logfn = d.getVar('BB_LOGFILE', True)
+                raise FuncFailed(func, logfn)
+        finally:
+            os.unlink(fifopath)
+
+    bb.debug(2, "Shell function %s finished" % func)
+
+def _task_data(fn, task, d):
+    localdata = bb.data.createCopy(d)
+    localdata.setVar('BB_FILENAME', fn)
+    localdata.setVar('BB_CURRENTTASK', task[3:])
+    localdata.setVar('OVERRIDES', 'task-%s:%s' %
+                     (task[3:].replace('_', '-'), d.getVar('OVERRIDES', False)))
+    localdata.finalize()
+    bb.data.expandKeys(localdata)
+    return localdata
+
+def _exec_task(fn, task, d, quieterr):
+    """Execute a BB 'task'
+
+    Execution of a task involves a bit more setup than executing a function,
+    running it with its own local metadata, and with some useful variables set.
+    """
+    if not d.getVarFlag(task, 'task', False):
+        event.fire(TaskInvalid(task, d), d)
+        logger.error("No such task: %s" % task)
+        return 1
+
+    logger.debug(1, "Executing task %s", task)
+
+    localdata = _task_data(fn, task, d)
+    tempdir = localdata.getVar('T', True)
+    if not tempdir:
+        bb.fatal("T variable not set, unable to build")
+
+    # Change nice level if we're asked to
+    nice = localdata.getVar("BB_TASK_NICE_LEVEL", True)
+    if nice:
+        curnice = os.nice(0)
+        nice = int(nice) - curnice
+        newnice = os.nice(nice)
+        logger.debug(1, "Renice to %s " % newnice)
+    ionice = localdata.getVar("BB_TASK_IONICE_LEVEL", True)
+    if ionice:
+        try:
+            cls, prio = ionice.split(".", 1)
+            bb.utils.ioprio_set(os.getpid(), int(cls), int(prio))
+        except:
+            bb.warn("Invalid ionice level %s" % ionice)
+
+    bb.utils.mkdirhier(tempdir)
+
+    # Determine the logfile to generate
+    logfmt = localdata.getVar('BB_LOGFMT', True) or 'log.{task}.{pid}'
+    logbase = logfmt.format(task=task, pid=os.getpid())
+
+    # Document the order of the tasks...
+    logorder = os.path.join(tempdir, 'log.task_order')
+    try:
+        with open(logorder, 'a') as logorderfile:
+            logorderfile.write('{0} ({1}): {2}\n'.format(task, os.getpid(), logbase))
+    except OSError:
+        logger.exception("Opening log file '%s'", logorder)
+        pass
+
+    # Setup the courtesy link to the logfn
+    loglink = os.path.join(tempdir, 'log.{0}'.format(task))
+    logfn = os.path.join(tempdir, logbase)
+    if loglink:
+        bb.utils.remove(loglink)
+
+        try:
+           os.symlink(logbase, loglink)
+        except OSError:
+           pass
+
+    prefuncs = localdata.getVarFlag(task, 'prefuncs', expand=True)
+    postfuncs = localdata.getVarFlag(task, 'postfuncs', expand=True)
+
+    class ErrorCheckHandler(logging.Handler):
+        def __init__(self):
+            self.triggered = False
+            logging.Handler.__init__(self, logging.ERROR)
+        def emit(self, record):
+            if getattr(record, 'forcelog', False):
+                self.triggered = False
+            else:
+                self.triggered = True
+
+    # Handle logfiles
+    si = open('/dev/null', 'r')
+    try:
+        bb.utils.mkdirhier(os.path.dirname(logfn))
+        logfile = open(logfn, 'w')
+    except OSError:
+        logger.exception("Opening log file '%s'", logfn)
+        pass
+
+    # Dup the existing fds so we dont lose them
+    osi = [os.dup(sys.stdin.fileno()), sys.stdin.fileno()]
+    oso = [os.dup(sys.stdout.fileno()), sys.stdout.fileno()]
+    ose = [os.dup(sys.stderr.fileno()), sys.stderr.fileno()]
+
+    # Replace those fds with our own
+    os.dup2(si.fileno(), osi[1])
+    os.dup2(logfile.fileno(), oso[1])
+    os.dup2(logfile.fileno(), ose[1])
+
+    # Ensure Python logging goes to the logfile
+    handler = logging.StreamHandler(logfile)
+    handler.setFormatter(logformatter)
+    # Always enable full debug output into task logfiles
+    handler.setLevel(logging.DEBUG - 2)
+    bblogger.addHandler(handler)
+
+    errchk = ErrorCheckHandler()
+    bblogger.addHandler(errchk)
+
+    localdata.setVar('BB_LOGFILE', logfn)
+    localdata.setVar('BB_RUNTASK', task)
+
+    flags = localdata.getVarFlags(task)
+
+    event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
+    try:
+        for func in (prefuncs or '').split():
+            exec_func(func, localdata)
+        exec_func(task, localdata)
+        for func in (postfuncs or '').split():
+            exec_func(func, localdata)
+    except FuncFailed as exc:
+        if quieterr:
+            event.fire(TaskFailedSilent(task, logfn, localdata), localdata)
+        else:
+            errprinted = errchk.triggered
+            logger.error(str(exc))
+            event.fire(TaskFailed(task, logfn, localdata, errprinted), localdata)
+        return 1
+    finally:
+        sys.stdout.flush()
+        sys.stderr.flush()
+
+        bblogger.removeHandler(handler)
+
+        # Restore the backup fds
+        os.dup2(osi[0], osi[1])
+        os.dup2(oso[0], oso[1])
+        os.dup2(ose[0], ose[1])
+
+        # Close the backup fds
+        os.close(osi[0])
+        os.close(oso[0])
+        os.close(ose[0])
+        si.close()
+
+        logfile.close()
+        if os.path.exists(logfn) and os.path.getsize(logfn) == 0:
+            logger.debug(2, "Zero size logfn %s, removing", logfn)
+            bb.utils.remove(logfn)
+            bb.utils.remove(loglink)
+    event.fire(TaskSucceeded(task, logfn, localdata), localdata)
+
+    if not localdata.getVarFlag(task, 'nostamp', False) and not localdata.getVarFlag(task, 'selfstamp', False):
+        make_stamp(task, localdata)
+
+    return 0
+
+def exec_task(fn, task, d, profile = False):
+    try:
+        quieterr = False
+        if d.getVarFlag(task, "quieterrors", False) is not None:
+            quieterr = True
+
+        if profile:
+            profname = "profile-%s.log" % (d.getVar("PN", True) + "-" + task)
+            try:
+                import cProfile as profile
+            except:
+                import profile
+            prof = profile.Profile()
+            ret = profile.Profile.runcall(prof, _exec_task, fn, task, d, quieterr)
+            prof.dump_stats(profname)
+            bb.utils.process_profilelog(profname)
+
+            return ret
+        else:
+            return _exec_task(fn, task, d, quieterr)
+
+    except Exception:
+        from traceback import format_exc
+        if not quieterr:
+            logger.error("Build of %s failed" % (task))
+            logger.error(format_exc())
+            failedevent = TaskFailed(task, None, d, True)
+            event.fire(failedevent, d)
+        return 1
+
+def stamp_internal(taskname, d, file_name, baseonly=False):
+    """
+    Internal stamp helper function
+    Makes sure the stamp directory exists
+    Returns the stamp path+filename
+
+    In the bitbake core, d can be a CacheData and file_name will be set.
+    When called in task context, d will be a data store, file_name will not be set
+    """
+    taskflagname = taskname
+    if taskname.endswith("_setscene") and taskname != "do_setscene":
+        taskflagname = taskname.replace("_setscene", "")
+
+    if file_name:
+        stamp = d.stamp[file_name]
+        extrainfo = d.stamp_extrainfo[file_name].get(taskflagname) or ""
+    else:
+        stamp = d.getVar('STAMP', True)
+        file_name = d.getVar('BB_FILENAME', True)
+        extrainfo = d.getVarFlag(taskflagname, 'stamp-extra-info', True) or ""
+
+    if baseonly:
+        return stamp
+
+    if not stamp:
+        return
+
+    stamp = bb.parse.siggen.stampfile(stamp, file_name, taskname, extrainfo)
+
+    stampdir = os.path.dirname(stamp)
+    if cached_mtime_noerror(stampdir) == 0:
+        bb.utils.mkdirhier(stampdir)
+
+    return stamp
+
+def stamp_cleanmask_internal(taskname, d, file_name):
+    """
+    Internal stamp helper function to generate stamp cleaning mask
+    Returns the stamp path+filename
+
+    In the bitbake core, d can be a CacheData and file_name will be set.
+    When called in task context, d will be a data store, file_name will not be set
+    """
+    taskflagname = taskname
+    if taskname.endswith("_setscene") and taskname != "do_setscene":
+        taskflagname = taskname.replace("_setscene", "")
+
+    if file_name:
+        stamp = d.stampclean[file_name]
+        extrainfo = d.stamp_extrainfo[file_name].get(taskflagname) or ""
+    else:
+        stamp = d.getVar('STAMPCLEAN', True)
+        file_name = d.getVar('BB_FILENAME', True)
+        extrainfo = d.getVarFlag(taskflagname, 'stamp-extra-info', True) or ""
+
+    if not stamp:
+        return []
+
+    cleanmask = bb.parse.siggen.stampcleanmask(stamp, file_name, taskname, extrainfo)
+
+    return [cleanmask, cleanmask.replace(taskflagname, taskflagname + "_setscene")]
+
+def make_stamp(task, d, file_name = None):
+    """
+    Creates/updates a stamp for a given task
+    (d can be a data dict or dataCache)
+    """
+    cleanmask = stamp_cleanmask_internal(task, d, file_name)
+    for mask in cleanmask:
+        for name in glob.glob(mask):
+            # Preserve sigdata files in the stamps directory
+            if "sigdata" in name:
+                continue
+            # Preserve taint files in the stamps directory
+            if name.endswith('.taint'):
+                continue
+            os.unlink(name)
+
+    stamp = stamp_internal(task, d, file_name)
+    # Remove the file and recreate to force timestamp
+    # change on broken NFS filesystems
+    if stamp:
+        bb.utils.remove(stamp)
+        open(stamp, "w").close()
+
+    # If we're in task context, write out a signature file for each task
+    # as it completes
+    if not task.endswith("_setscene") and task != "do_setscene" and not file_name:
+        stampbase = stamp_internal(task, d, None, True)
+        file_name = d.getVar('BB_FILENAME', True)
+        bb.parse.siggen.dump_sigtask(file_name, task, stampbase, True)
+
+def del_stamp(task, d, file_name = None):
+    """
+    Removes a stamp for a given task
+    (d can be a data dict or dataCache)
+    """
+    stamp = stamp_internal(task, d, file_name)
+    bb.utils.remove(stamp)
+
+def write_taint(task, d, file_name = None):
+    """
+    Creates a "taint" file which will force the specified task and its
+    dependents to be re-run the next time by influencing the value of its
+    taskhash.
+    (d can be a data dict or dataCache)
+    """
+    import uuid
+    if file_name:
+        taintfn = d.stamp[file_name] + '.' + task + '.taint'
+    else:
+        taintfn = d.getVar('STAMP', True) + '.' + task + '.taint'
+    bb.utils.mkdirhier(os.path.dirname(taintfn))
+    # The specific content of the taint file is not really important,
+    # we just need it to be random, so a random UUID is used
+    with open(taintfn, 'w') as taintf:
+        taintf.write(str(uuid.uuid4()))
+
+def stampfile(taskname, d, file_name = None):
+    """
+    Return the stamp for a given task
+    (d can be a data dict or dataCache)
+    """
+    return stamp_internal(taskname, d, file_name)
+
+def add_tasks(tasklist, d):
+    task_deps = d.getVar('_task_deps', False)
+    if not task_deps:
+        task_deps = {}
+    if not 'tasks' in task_deps:
+        task_deps['tasks'] = []
+    if not 'parents' in task_deps:
+        task_deps['parents'] = {}
+
+    for task in tasklist:
+        task = d.expand(task)
+
+        d.setVarFlag(task, 'task', 1)
+
+        if not task in task_deps['tasks']:
+            task_deps['tasks'].append(task)
+
+        flags = d.getVarFlags(task)
+        def getTask(name):
+            if not name in task_deps:
+                task_deps[name] = {}
+            if name in flags:
+                deptask = d.expand(flags[name])
+                task_deps[name][task] = deptask
+        getTask('depends')
+        getTask('rdepends')
+        getTask('deptask')
+        getTask('rdeptask')
+        getTask('recrdeptask')
+        getTask('recideptask')
+        getTask('nostamp')
+        getTask('fakeroot')
+        getTask('noexec')
+        getTask('umask')
+        task_deps['parents'][task] = []
+        if 'deps' in flags:
+            for dep in flags['deps']:
+                dep = d.expand(dep)
+                task_deps['parents'][task].append(dep)
+
+    # don't assume holding a reference
+    d.setVar('_task_deps', task_deps)
+
+def addtask(task, before, after, d):
+    if task[:3] != "do_":
+        task = "do_" + task
+
+    d.setVarFlag(task, "task", 1)
+    bbtasks = d.getVar('__BBTASKS', False) or []
+    if task not in bbtasks:
+        bbtasks.append(task)
+    d.setVar('__BBTASKS', bbtasks)
+
+    existing = d.getVarFlag(task, "deps", False) or []
+    if after is not None:
+        # set up deps for function
+        for entry in after.split():
+            if entry not in existing:
+                existing.append(entry)
+    d.setVarFlag(task, "deps", existing)
+    if before is not None:
+        # set up things that depend on this func
+        for entry in before.split():
+            existing = d.getVarFlag(entry, "deps", False) or []
+            if task not in existing:
+                d.setVarFlag(entry, "deps", [task] + existing)
+
+def deltask(task, d):
+    if task[:3] != "do_":
+        task = "do_" + task
+
+    bbtasks = d.getVar('__BBTASKS', False) or []
+    if task in bbtasks:
+        bbtasks.remove(task)
+        d.setVar('__BBTASKS', bbtasks)
+
+    d.delVarFlag(task, 'deps')
+    for bbtask in d.getVar('__BBTASKS', False) or []:
+        deps = d.getVarFlag(bbtask, 'deps', False) or []
+        if task in deps:
+            deps.remove(task)
+            d.setVarFlag(bbtask, 'deps', deps)

bitbake/lib/bb/cache.py

@@ -0,0 +1,849 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# BitBake Cache implementation
+#
+# Caching of bitbake variables before task execution
+
+# Copyright (C) 2006        Richard Purdie
+# Copyright (C) 2012        Intel Corporation
+
+# but small sections based on code from bin/bitbake:
+# Copyright (C) 2003, 2004  Chris Larson
+# Copyright (C) 2003, 2004  Phil Blundell
+# Copyright (C) 2003 - 2005 Michael 'Mickey' Lauer
+# Copyright (C) 2005        Holger Hans Peter Freyther
+# Copyright (C) 2005        ROAD GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+
+import os
+import logging
+from collections import defaultdict
+import bb.utils
+
+logger = logging.getLogger("BitBake.Cache")
+
+try:
+    import cPickle as pickle
+except ImportError:
+    import pickle
+    logger.info("Importing cPickle failed. "
+                "Falling back to a very slow implementation.")
+
+__cache_version__ = "149"
+
+def getCacheFile(path, filename, data_hash):
+    return os.path.join(path, filename + "." + data_hash)
+
+# RecipeInfoCommon defines common data retrieving methods
+# from meta data for caches. CoreRecipeInfo as well as other
+# Extra RecipeInfo needs to inherit this class
+class RecipeInfoCommon(object):
+
+    @classmethod
+    def listvar(cls, var, metadata):
+        return cls.getvar(var, metadata).split()
+
+    @classmethod
+    def intvar(cls, var, metadata):
+        return int(cls.getvar(var, metadata) or 0)
+
+    @classmethod
+    def depvar(cls, var, metadata):
+        return bb.utils.explode_deps(cls.getvar(var, metadata))
+
+    @classmethod
+    def pkgvar(cls, var, packages, metadata):
+        return dict((pkg, cls.depvar("%s_%s" % (var, pkg), metadata))
+                    for pkg in packages)
+
+    @classmethod
+    def taskvar(cls, var, tasks, metadata):
+        return dict((task, cls.getvar("%s_task-%s" % (var, task), metadata))
+                    for task in tasks)
+
+    @classmethod
+    def flaglist(cls, flag, varlist, metadata, squash=False):
+        out_dict = dict((var, metadata.getVarFlag(var, flag, True))
+                    for var in varlist)
+        if squash:
+            return dict((k,v) for (k,v) in out_dict.iteritems() if v)
+        else:
+            return out_dict
+
+    @classmethod
+    def getvar(cls, var, metadata, expand = True):
+        return metadata.getVar(var, expand) or ''
+
+
+class CoreRecipeInfo(RecipeInfoCommon):
+    __slots__ = ()
+
+    cachefile = "bb_cache.dat"   
+
+    def __init__(self, filename, metadata):      
+        self.file_depends = metadata.getVar('__depends', False)
+        self.timestamp = bb.parse.cached_mtime(filename)
+        self.variants = self.listvar('__VARIANTS', metadata) + ['']
+        self.appends = self.listvar('__BBAPPEND', metadata)
+        self.nocache = self.getvar('BB_DONT_CACHE', metadata)
+
+        self.skipreason = self.getvar('__SKIPPED', metadata)
+        if self.skipreason:
+            self.pn = self.getvar('PN', metadata) or bb.parse.BBHandler.vars_from_file(filename,metadata)[0]
+            self.skipped = True
+            self.provides  = self.depvar('PROVIDES', metadata)
+            self.rprovides = self.depvar('RPROVIDES', metadata)
+            return
+
+        self.tasks = metadata.getVar('__BBTASKS', False)
+
+        self.pn = self.getvar('PN', metadata)
+        self.packages = self.listvar('PACKAGES', metadata)
+        if not self.pn in self.packages:
+            self.packages.append(self.pn)
+
+        self.basetaskhashes = self.taskvar('BB_BASEHASH', self.tasks, metadata)
+        self.hashfilename = self.getvar('BB_HASHFILENAME', metadata)
+
+        self.task_deps = metadata.getVar('_task_deps', False) or {'tasks': [], 'parents': {}}
+
+        self.skipped = False
+        self.pe = self.getvar('PE', metadata)
+        self.pv = self.getvar('PV', metadata)
+        self.pr = self.getvar('PR', metadata)
+        self.defaultpref = self.intvar('DEFAULT_PREFERENCE', metadata)
+        self.not_world = self.getvar('EXCLUDE_FROM_WORLD', metadata)
+        self.stamp = self.getvar('STAMP', metadata)
+        self.stampclean = self.getvar('STAMPCLEAN', metadata)        
+        self.stamp_extrainfo = self.flaglist('stamp-extra-info', self.tasks, metadata)
+        self.file_checksums = self.flaglist('file-checksums', self.tasks, metadata, True)
+        self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC', metadata)
+        self.depends          = self.depvar('DEPENDS', metadata)
+        self.provides         = self.depvar('PROVIDES', metadata)
+        self.rdepends         = self.depvar('RDEPENDS', metadata)
+        self.rprovides        = self.depvar('RPROVIDES', metadata)
+        self.rrecommends      = self.depvar('RRECOMMENDS', metadata)
+        self.rprovides_pkg    = self.pkgvar('RPROVIDES', self.packages, metadata)
+        self.rdepends_pkg     = self.pkgvar('RDEPENDS', self.packages, metadata)
+        self.rrecommends_pkg  = self.pkgvar('RRECOMMENDS', self.packages, metadata)
+        self.inherits         = self.getvar('__inherit_cache', metadata, expand=False)
+        self.fakerootenv      = self.getvar('FAKEROOTENV', metadata)
+        self.fakerootdirs     = self.getvar('FAKEROOTDIRS', metadata)
+        self.fakerootnoenv    = self.getvar('FAKEROOTNOENV', metadata)
+        self.extradepsfunc    = self.getvar('calculate_extra_depends', metadata)
+
+    @classmethod
+    def init_cacheData(cls, cachedata):
+        # CacheData in Core RecipeInfo Class
+        cachedata.task_deps = {}
+        cachedata.pkg_fn = {}
+        cachedata.pkg_pn = defaultdict(list)
+        cachedata.pkg_pepvpr = {}
+        cachedata.pkg_dp = {}
+
+        cachedata.stamp = {}
+        cachedata.stampclean = {}
+        cachedata.stamp_extrainfo = {}
+        cachedata.file_checksums = {}
+        cachedata.fn_provides = {}
+        cachedata.pn_provides = defaultdict(list)
+        cachedata.all_depends = []
+
+        cachedata.deps = defaultdict(list)
+        cachedata.packages = defaultdict(list)
+        cachedata.providers = defaultdict(list)
+        cachedata.rproviders = defaultdict(list)
+        cachedata.packages_dynamic = defaultdict(list)
+
+        cachedata.rundeps = defaultdict(lambda: defaultdict(list))
+        cachedata.runrecs = defaultdict(lambda: defaultdict(list))
+        cachedata.possible_world = []
+        cachedata.universe_target = []
+        cachedata.hashfn = {}
+
+        cachedata.basetaskhash = {}
+        cachedata.inherits = {}
+        cachedata.fakerootenv = {}
+        cachedata.fakerootnoenv = {}
+        cachedata.fakerootdirs = {}
+        cachedata.extradepsfunc = {}
+
+    def add_cacheData(self, cachedata, fn):
+        cachedata.task_deps[fn] = self.task_deps
+        cachedata.pkg_fn[fn] = self.pn
+        cachedata.pkg_pn[self.pn].append(fn)
+        cachedata.pkg_pepvpr[fn] = (self.pe, self.pv, self.pr)
+        cachedata.pkg_dp[fn] = self.defaultpref
+        cachedata.stamp[fn] = self.stamp
+        cachedata.stampclean[fn] = self.stampclean
+        cachedata.stamp_extrainfo[fn] = self.stamp_extrainfo
+        cachedata.file_checksums[fn] = self.file_checksums
+
+        provides = [self.pn]
+        for provide in self.provides:
+            if provide not in provides:
+                provides.append(provide)
+        cachedata.fn_provides[fn] = provides
+
+        for provide in provides:
+            cachedata.providers[provide].append(fn)
+            if provide not in cachedata.pn_provides[self.pn]:
+                cachedata.pn_provides[self.pn].append(provide)
+
+        for dep in self.depends:
+            if dep not in cachedata.deps[fn]:
+                cachedata.deps[fn].append(dep)
+            if dep not in cachedata.all_depends:
+                cachedata.all_depends.append(dep)
+
+        rprovides = self.rprovides
+        for package in self.packages:
+            cachedata.packages[package].append(fn)
+            rprovides += self.rprovides_pkg[package]
+
+        for rprovide in rprovides:
+            if fn not in cachedata.rproviders[rprovide]:
+                cachedata.rproviders[rprovide].append(fn)
+
+        for package in self.packages_dynamic:
+            cachedata.packages_dynamic[package].append(fn)
+
+        # Build hash of runtime depends and recommends
+        for package in self.packages + [self.pn]:
+            cachedata.rundeps[fn][package] = list(self.rdepends) + self.rdepends_pkg[package]
+            cachedata.runrecs[fn][package] = list(self.rrecommends) + self.rrecommends_pkg[package]
+
+        # Collect files we may need for possible world-dep
+        # calculations
+        if self.not_world:
+            logger.debug(1, "EXCLUDE FROM WORLD: %s", fn)
+        else:
+            cachedata.possible_world.append(fn)
+
+        # create a collection of all targets for sanity checking
+        # tasks, such as upstream versions, license, and tools for
+        # task and image creation.
+        cachedata.universe_target.append(self.pn)
+
+        cachedata.hashfn[fn] = self.hashfilename
+        for task, taskhash in self.basetaskhashes.iteritems():
+            identifier = '%s.%s' % (fn, task)
+            cachedata.basetaskhash[identifier] = taskhash
+
+        cachedata.inherits[fn] = self.inherits
+        cachedata.fakerootenv[fn] = self.fakerootenv
+        cachedata.fakerootnoenv[fn] = self.fakerootnoenv
+        cachedata.fakerootdirs[fn] = self.fakerootdirs
+        cachedata.extradepsfunc[fn] = self.extradepsfunc
+
+
+
+class Cache(object):
+    """
+    BitBake Cache implementation
+    """
+
+    def __init__(self, data, data_hash, caches_array):
+        # Pass caches_array information into Cache Constructor
+        # It will be used later for deciding whether we 
+        # need extra cache file dump/load support 
+        self.caches_array = caches_array
+        self.cachedir = data.getVar("CACHE", True)
+        self.clean = set()
+        self.checked = set()
+        self.depends_cache = {}
+        self.data = None
+        self.data_fn = None
+        self.cacheclean = True
+        self.data_hash = data_hash
+
+        if self.cachedir in [None, '']:
+            self.has_cache = False
+            logger.info("Not using a cache. "
+                        "Set CACHE = <directory> to enable.")
+            return
+
+        self.has_cache = True
+        self.cachefile = getCacheFile(self.cachedir, "bb_cache.dat", self.data_hash)
+
+        logger.debug(1, "Using cache in '%s'", self.cachedir)
+        bb.utils.mkdirhier(self.cachedir)
+
+        cache_ok = True
+        if self.caches_array:
+            for cache_class in self.caches_array:
+                if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
+                    cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
+                    cache_ok = cache_ok and os.path.exists(cachefile)
+                    cache_class.init_cacheData(self)
+        if cache_ok:
+            self.load_cachefile()
+        elif os.path.isfile(self.cachefile):
+            logger.info("Out of date cache found, rebuilding...")
+
+    def load_cachefile(self):
+        # Firstly, using core cache file information for
+        # valid checking
+        with open(self.cachefile, "rb") as cachefile:
+            pickled = pickle.Unpickler(cachefile)
+            try:
+                cache_ver = pickled.load()
+                bitbake_ver = pickled.load()
+            except Exception:
+                logger.info('Invalid cache, rebuilding...')
+                return
+
+            if cache_ver != __cache_version__:
+                logger.info('Cache version mismatch, rebuilding...')
+                return
+            elif bitbake_ver != bb.__version__:
+                logger.info('Bitbake version mismatch, rebuilding...')
+                return
+
+
+        cachesize = 0
+        previous_progress = 0
+        previous_percent = 0
+
+        # Calculate the correct cachesize of all those cache files
+        for cache_class in self.caches_array:
+            if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
+                cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
+                with open(cachefile, "rb") as cachefile:
+                    cachesize += os.fstat(cachefile.fileno()).st_size
+
+        bb.event.fire(bb.event.CacheLoadStarted(cachesize), self.data)
+        
+        for cache_class in self.caches_array:
+            if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
+                cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
+                with open(cachefile, "rb") as cachefile:
+                    pickled = pickle.Unpickler(cachefile)                    
+                    while cachefile:
+                        try:
+                            key = pickled.load()
+                            value = pickled.load()
+                        except Exception:
+                            break
+                        if self.depends_cache.has_key(key):
+                            self.depends_cache[key].append(value)
+                        else:
+                            self.depends_cache[key] = [value]
+                        # only fire events on even percentage boundaries
+                        current_progress = cachefile.tell() + previous_progress
+                        current_percent = 100 * current_progress / cachesize
+                        if current_percent > previous_percent:
+                            previous_percent = current_percent
+                            bb.event.fire(bb.event.CacheLoadProgress(current_progress, cachesize),
+                                          self.data)
+
+                    previous_progress += current_progress
+
+        # Note: depends cache number is corresponding to the parsing file numbers.
+        # The same file has several caches, still regarded as one item in the cache
+        bb.event.fire(bb.event.CacheLoadCompleted(cachesize,
+                                                  len(self.depends_cache)),
+                      self.data)
+
+    
+    @staticmethod
+    def virtualfn2realfn(virtualfn):
+        """
+        Convert a virtual file name to a real one + the associated subclass keyword
+        """
+
+        fn = virtualfn
+        cls = ""
+        if virtualfn.startswith('virtual:'):
+            elems = virtualfn.split(':')
+            cls = ":".join(elems[1:-1])
+            fn = elems[-1]
+        return (fn, cls)
+
+    @staticmethod
+    def realfn2virtual(realfn, cls):
+        """
+        Convert a real filename + the associated subclass keyword to a virtual filename
+        """
+        if cls == "":
+            return realfn
+        return "virtual:" + cls + ":" + realfn
+
+    @classmethod
+    def loadDataFull(cls, virtualfn, appends, cfgData):
+        """
+        Return a complete set of data for fn.
+        To do this, we need to parse the file.
+        """
+
+        (fn, virtual) = cls.virtualfn2realfn(virtualfn)
+
+        logger.debug(1, "Parsing %s (full)", fn)
+
+        cfgData.setVar("__ONLYFINALISE", virtual or "default")
+        bb_data = cls.load_bbfile(fn, appends, cfgData)
+        return bb_data[virtual]
+
+    @classmethod
+    def parse(cls, filename, appends, configdata, caches_array):
+        """Parse the specified filename, returning the recipe information"""
+        infos = []
+        datastores = cls.load_bbfile(filename, appends, configdata)
+        depends = []
+        for variant, data in sorted(datastores.iteritems(),
+                                    key=lambda i: i[0],
+                                    reverse=True):
+            virtualfn = cls.realfn2virtual(filename, variant)
+            depends = depends + (data.getVar("__depends", False) or [])
+            if depends and not variant:
+                data.setVar("__depends", depends)
+
+            info_array = []
+            for cache_class in caches_array:
+                if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
+                    info = cache_class(filename, data)
+                    info_array.append(info)
+            infos.append((virtualfn, info_array))
+
+        return infos
+
+    def load(self, filename, appends, configdata):
+        """Obtain the recipe information for the specified filename,
+        using cached values if available, otherwise parsing.
+
+        Note that if it does parse to obtain the info, it will not
+        automatically add the information to the cache or to your
+        CacheData.  Use the add or add_info method to do so after
+        running this, or use loadData instead."""
+        cached = self.cacheValid(filename, appends)
+        if cached:
+            infos = []
+            # info_array item is a list of [CoreRecipeInfo, XXXRecipeInfo]
+            info_array = self.depends_cache[filename]
+            for variant in info_array[0].variants:
+                virtualfn = self.realfn2virtual(filename, variant)
+                infos.append((virtualfn, self.depends_cache[virtualfn]))
+        else:
+            logger.debug(1, "Parsing %s", filename)
+            return self.parse(filename, appends, configdata, self.caches_array)
+
+        return cached, infos
+
+    def loadData(self, fn, appends, cfgData, cacheData):
+        """Load the recipe info for the specified filename,
+        parsing and adding to the cache if necessary, and adding
+        the recipe information to the supplied CacheData instance."""
+        skipped, virtuals = 0, 0
+
+        cached, infos = self.load(fn, appends, cfgData)
+        for virtualfn, info_array in infos:
+            if info_array[0].skipped:
+                logger.debug(1, "Skipping %s: %s", virtualfn, info_array[0].skipreason)
+                skipped += 1
+            else:
+                self.add_info(virtualfn, info_array, cacheData, not cached)
+                virtuals += 1
+
+        return cached, skipped, virtuals
+
+    def cacheValid(self, fn, appends):
+        """
+        Is the cache valid for fn?
+        Fast version, no timestamps checked.
+        """
+        if fn not in self.checked:
+            self.cacheValidUpdate(fn, appends)
+
+        # Is cache enabled?
+        if not self.has_cache:
+            return False
+        if fn in self.clean:
+            return True
+        return False
+
+    def cacheValidUpdate(self, fn, appends):
+        """
+        Is the cache valid for fn?
+        Make thorough (slower) checks including timestamps.
+        """
+        # Is cache enabled?
+        if not self.has_cache:
+            return False
+
+        self.checked.add(fn)
+
+        # File isn't in depends_cache
+        if not fn in self.depends_cache:
+            logger.debug(2, "Cache: %s is not cached", fn)
+            return False
+
+        mtime = bb.parse.cached_mtime_noerror(fn)
+
+        # Check file still exists
+        if mtime == 0:
+            logger.debug(2, "Cache: %s no longer exists", fn)
+            self.remove(fn)
+            return False
+
+        info_array = self.depends_cache[fn]
+        # Check the file's timestamp
+        if mtime != info_array[0].timestamp:
+            logger.debug(2, "Cache: %s changed", fn)
+            self.remove(fn)
+            return False
+
+        # Check dependencies are still valid
+        depends = info_array[0].file_depends
+        if depends:
+            for f, old_mtime in depends:
+                fmtime = bb.parse.cached_mtime_noerror(f)
+                # Check if file still exists
+                if old_mtime != 0 and fmtime == 0:
+                    logger.debug(2, "Cache: %s's dependency %s was removed",
+                                    fn, f)
+                    self.remove(fn)
+                    return False
+
+                if (fmtime != old_mtime):
+                    logger.debug(2, "Cache: %s's dependency %s changed",
+                                    fn, f)
+                    self.remove(fn)
+                    return False
+
+        if hasattr(info_array[0], 'file_checksums'):
+            for _, fl in info_array[0].file_checksums.items():
+                fl = fl.strip()
+                while fl:
+                    # A .split() would be simpler but means spaces or colons in filenames would break
+                    a = fl.find(":True")
+                    b = fl.find(":False")
+                    if ((a < 0) and b) or ((b > 0) and (b < a)):
+                       f = fl[:b+6]
+                       fl = fl[b+7:]
+                    elif ((b < 0) and a) or ((a > 0) and (a < b)):
+                       f = fl[:a+5]
+                       fl = fl[a+6:]
+                    else:
+                       break
+                    fl = fl.strip()
+                    if "*" in f:
+                        continue
+                    f, exist = f.split(":")
+                    if (exist == "True" and not os.path.exists(f)) or (exist == "False" and os.path.exists(f)):
+                        logger.debug(2, "Cache: %s's file checksum list file %s changed",
+                                        fn, f)
+                        self.remove(fn)
+                        return False
+
+        if appends != info_array[0].appends:
+            logger.debug(2, "Cache: appends for %s changed", fn)
+            logger.debug(2, "%s to %s" % (str(appends), str(info_array[0].appends)))
+            self.remove(fn)
+            return False
+
+        invalid = False
+        for cls in info_array[0].variants:
+            virtualfn = self.realfn2virtual(fn, cls)
+            self.clean.add(virtualfn)
+            if virtualfn not in self.depends_cache:
+                logger.debug(2, "Cache: %s is not cached", virtualfn)
+                invalid = True
+
+        # If any one of the variants is not present, mark as invalid for all
+        if invalid:
+            for cls in info_array[0].variants:
+                virtualfn = self.realfn2virtual(fn, cls)
+                if virtualfn in self.clean:
+                    logger.debug(2, "Cache: Removing %s from cache", virtualfn)
+                    self.clean.remove(virtualfn)
+            if fn in self.clean:
+                logger.debug(2, "Cache: Marking %s as not clean", fn)
+                self.clean.remove(fn)
+            return False
+
+        self.clean.add(fn)
+        return True
+
+    def remove(self, fn):
+        """
+        Remove a fn from the cache
+        Called from the parser in error cases
+        """
+        if fn in self.depends_cache:
+            logger.debug(1, "Removing %s from cache", fn)
+            del self.depends_cache[fn]
+        if fn in self.clean:
+            logger.debug(1, "Marking %s as unclean", fn)
+            self.clean.remove(fn)
+
+    def sync(self):
+        """
+        Save the cache
+        Called from the parser when complete (or exiting)
+        """
+
+        if not self.has_cache:
+            return
+
+        if self.cacheclean:
+            logger.debug(2, "Cache is clean, not saving.")
+            return
+
+        file_dict = {}
+        pickler_dict = {}
+        for cache_class in self.caches_array:
+            if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
+                cache_class_name = cache_class.__name__
+                cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
+                file_dict[cache_class_name] = open(cachefile, "wb")
+                pickler_dict[cache_class_name] =  pickle.Pickler(file_dict[cache_class_name], pickle.HIGHEST_PROTOCOL)
+                   
+        pickler_dict['CoreRecipeInfo'].dump(__cache_version__)
+        pickler_dict['CoreRecipeInfo'].dump(bb.__version__)
+
+        try:
+            for key, info_array in self.depends_cache.iteritems():
+                for info in info_array:
+                    if isinstance(info, RecipeInfoCommon):
+                        cache_class_name = info.__class__.__name__
+                        pickler_dict[cache_class_name].dump(key)
+                        pickler_dict[cache_class_name].dump(info)
+        finally:
+            for cache_class in self.caches_array:
+                if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
+                    cache_class_name = cache_class.__name__
+                    file_dict[cache_class_name].close()
+
+        del self.depends_cache
+
+    @staticmethod
+    def mtime(cachefile):
+        return bb.parse.cached_mtime_noerror(cachefile)
+
+    def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None):
+        if isinstance(info_array[0], CoreRecipeInfo) and (not info_array[0].skipped):
+            cacheData.add_from_recipeinfo(filename, info_array)
+
+            if watcher:
+                watcher(info_array[0].file_depends)
+
+        if not self.has_cache:
+            return
+
+        if (info_array[0].skipped or 'SRCREVINACTION' not in info_array[0].pv) and not info_array[0].nocache:
+            if parsed:
+                self.cacheclean = False
+            self.depends_cache[filename] = info_array
+
+    def add(self, file_name, data, cacheData, parsed=None):
+        """
+        Save data we need into the cache
+        """
+
+        realfn = self.virtualfn2realfn(file_name)[0]
+
+        info_array = []
+        for cache_class in self.caches_array:
+            if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
+                info_array.append(cache_class(realfn, data))
+        self.add_info(file_name, info_array, cacheData, parsed)
+
+    @staticmethod
+    def load_bbfile(bbfile, appends, config):
+        """
+        Load and parse one .bb build file
+        Return the data and whether parsing resulted in the file being skipped
+        """
+        chdir_back = False
+
+        from bb import parse
+
+        # expand tmpdir to include this topdir
+        config.setVar('TMPDIR', config.getVar('TMPDIR', True) or "")
+        bbfile_loc = os.path.abspath(os.path.dirname(bbfile))
+        oldpath = os.path.abspath(os.getcwd())
+        parse.cached_mtime_noerror(bbfile_loc)
+        bb_data = config.createCopy()
+        # The ConfHandler first looks if there is a TOPDIR and if not
+        # then it would call getcwd().
+        # Previously, we chdir()ed to bbfile_loc, called the handler
+        # and finally chdir()ed back, a couple of thousand times. We now
+        # just fill in TOPDIR to point to bbfile_loc if there is no TOPDIR yet.
+        if not bb_data.getVar('TOPDIR', False):
+            chdir_back = True
+            bb_data.setVar('TOPDIR', bbfile_loc)
+        try:
+            if appends:
+                bb_data.setVar('__BBAPPEND', " ".join(appends))
+            bb_data = parse.handle(bbfile, bb_data)
+            if chdir_back:
+                os.chdir(oldpath)
+            return bb_data
+        except:
+            if chdir_back:
+                os.chdir(oldpath)
+            raise
+
+
+def init(cooker):
+    """
+    The Objective: Cache the minimum amount of data possible yet get to the
+    stage of building packages (i.e. tryBuild) without reparsing any .bb files.
+
+    To do this, we intercept getVar calls and only cache the variables we see
+    being accessed. We rely on the cache getVar calls being made for all
+    variables bitbake might need to use to reach this stage. For each cached
+    file we need to track:
+
+    * Its mtime
+    * The mtimes of all its dependencies
+    * Whether it caused a parse.SkipRecipe exception
+
+    Files causing parsing errors are evicted from the cache.
+
+    """
+    return Cache(cooker.configuration.data, cooker.configuration.data_hash)
+
+
+class CacheData(object):
+    """
+    The data structures we compile from the cached data
+    """
+
+    def __init__(self, caches_array):
+        self.caches_array = caches_array
+        for cache_class in self.caches_array:
+            if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
+                cache_class.init_cacheData(self)        
+
+        # Direct cache variables
+        self.task_queues = {}
+        self.preferred = {}
+        self.tasks = {}
+        # Indirect Cache variables (set elsewhere)
+        self.ignored_dependencies = []
+        self.world_target = set()
+        self.bbfile_priority = {}
+
+    def add_from_recipeinfo(self, fn, info_array):
+        for info in info_array:
+            info.add_cacheData(self, fn)
+
+class MultiProcessCache(object):
+    """
+    BitBake multi-process cache implementation
+
+    Used by the codeparser & file checksum caches
+    """
+
+    def __init__(self):
+        self.cachefile = None
+        self.cachedata = self.create_cachedata()
+        self.cachedata_extras = self.create_cachedata()
+
+    def init_cache(self, d, cache_file_name=None):
+        cachedir = (d.getVar("PERSISTENT_DIR", True) or
+                    d.getVar("CACHE", True))
+        if cachedir in [None, '']:
+            return
+        bb.utils.mkdirhier(cachedir)
+        self.cachefile = os.path.join(cachedir,
+                                      cache_file_name or self.__class__.cache_file_name)
+        logger.debug(1, "Using cache in '%s'", self.cachefile)
+
+        glf = bb.utils.lockfile(self.cachefile + ".lock")
+
+        try:
+            with open(self.cachefile, "rb") as f:
+                p = pickle.Unpickler(f)
+                data, version = p.load()
+        except:
+            bb.utils.unlockfile(glf)
+            return
+
+        bb.utils.unlockfile(glf)
+
+        if version != self.__class__.CACHE_VERSION:
+            return
+
+        self.cachedata = data
+
+    def create_cachedata(self):
+        data = [{}]
+        return data
+
+    def save_extras(self):
+        if not self.cachefile:
+            return
+
+        glf = bb.utils.lockfile(self.cachefile + ".lock", shared=True)
+
+        i = os.getpid()
+        lf = None
+        while not lf:
+            lf = bb.utils.lockfile(self.cachefile + ".lock." + str(i), retry=False)
+            if not lf or os.path.exists(self.cachefile + "-" + str(i)):
+                if lf:
+                    bb.utils.unlockfile(lf)
+                    lf = None
+                i = i + 1
+                continue
+
+            with open(self.cachefile + "-" + str(i), "wb") as f:
+                p = pickle.Pickler(f, -1)
+                p.dump([self.cachedata_extras, self.__class__.CACHE_VERSION])
+
+        bb.utils.unlockfile(lf)
+        bb.utils.unlockfile(glf)
+
+    def merge_data(self, source, dest):
+        for j in range(0,len(dest)):
+            for h in source[j]:
+                if h not in dest[j]:
+                    dest[j][h] = source[j][h]
+
+    def save_merge(self):
+        if not self.cachefile:
+            return
+
+        glf = bb.utils.lockfile(self.cachefile + ".lock")
+
+        data = self.cachedata
+
+        for f in [y for y in os.listdir(os.path.dirname(self.cachefile)) if y.startswith(os.path.basename(self.cachefile) + '-')]:
+            f = os.path.join(os.path.dirname(self.cachefile), f)
+            try:
+                with open(f, "rb") as fd:
+                    p = pickle.Unpickler(fd)
+                    extradata, version = p.load()
+            except (IOError, EOFError):
+                os.unlink(f)
+                continue
+
+            if version != self.__class__.CACHE_VERSION:
+                os.unlink(f)
+                continue
+
+            self.merge_data(extradata, data)
+            os.unlink(f)
+
+        with open(self.cachefile, "wb") as f:
+            p = pickle.Pickler(f, -1)
+            p.dump([data, self.__class__.CACHE_VERSION])
+
+        bb.utils.unlockfile(glf)
+

bitbake/lib/bb/cache_extra.py

@@ -0,0 +1,75 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Extra RecipeInfo will be all defined in this file. Currently,
+# Only Hob (Image Creator) Requests some extra fields. So
+# HobRecipeInfo is defined. It's named HobRecipeInfo because it
+# is introduced by 'hob'. Users could also introduce other
+# RecipeInfo or simply use those already defined RecipeInfo.
+# In the following patch, this newly defined new extra RecipeInfo
+# will be dynamically loaded and used for loading/saving the extra
+# cache fields  
+
+# Copyright (C) 2011, Intel Corporation. All rights reserved.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from bb.cache import RecipeInfoCommon
+
+class HobRecipeInfo(RecipeInfoCommon):
+    __slots__ = ()
+
+    classname = "HobRecipeInfo"
+    # please override this member with the correct data cache file
+    # such as (bb_cache.dat, bb_extracache_hob.dat) 
+    cachefile = "bb_extracache_" + classname +".dat"        
+
+    # override this member with the list of extra cache fields
+    # that this class will provide
+    cachefields = ['summary', 'license', 'section',
+            'description', 'homepage', 'bugtracker',
+            'prevision', 'files_info']
+
+    def __init__(self, filename, metadata):
+
+        self.summary = self.getvar('SUMMARY', metadata)
+        self.license = self.getvar('LICENSE', metadata)
+        self.section = self.getvar('SECTION', metadata)
+        self.description = self.getvar('DESCRIPTION', metadata)
+        self.homepage = self.getvar('HOMEPAGE', metadata)
+        self.bugtracker = self.getvar('BUGTRACKER', metadata)
+        self.prevision = self.getvar('PR', metadata)
+        self.files_info = self.getvar('FILES_INFO', metadata)
+
+    @classmethod
+    def init_cacheData(cls, cachedata):
+        # CacheData in Hob RecipeInfo Class
+        cachedata.summary = {}
+        cachedata.license = {}
+        cachedata.section = {}
+        cachedata.description = {}
+        cachedata.homepage = {}
+        cachedata.bugtracker = {}
+        cachedata.prevision = {}
+        cachedata.files_info = {}
+
+    def add_cacheData(self, cachedata, fn):
+        cachedata.summary[fn] = self.summary
+        cachedata.license[fn] = self.license
+        cachedata.section[fn] = self.section
+        cachedata.description[fn] = self.description
+        cachedata.homepage[fn] = self.homepage
+        cachedata.bugtracker[fn] = self.bugtracker
+        cachedata.prevision[fn] = self.prevision
+        cachedata.files_info[fn] = self.files_info

bitbake/lib/bb/checksum.py

@@ -0,0 +1,139 @@
+# Local file checksum cache implementation
+#
+# Copyright (C) 2012 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import glob
+import operator
+import os
+import stat
+import bb.utils
+import logging
+from bb.cache import MultiProcessCache
+
+logger = logging.getLogger("BitBake.Cache")
+
+try:
+    import cPickle as pickle
+except ImportError:
+    import pickle
+    logger.info("Importing cPickle failed. "
+                "Falling back to a very slow implementation.")
+
+
+# mtime cache (non-persistent)
+# based upon the assumption that files do not change during bitbake run
+class FileMtimeCache(object):
+    cache = {}
+
+    def cached_mtime(self, f):
+        if f not in self.cache:
+            self.cache[f] = os.stat(f)[stat.ST_MTIME]
+        return self.cache[f]
+
+    def cached_mtime_noerror(self, f):
+        if f not in self.cache:
+            try:
+                self.cache[f] = os.stat(f)[stat.ST_MTIME]
+            except OSError:
+                return 0
+        return self.cache[f]
+
+    def update_mtime(self, f):
+        self.cache[f] = os.stat(f)[stat.ST_MTIME]
+        return self.cache[f]
+
+    def clear(self):
+        self.cache.clear()
+
+# Checksum + mtime cache (persistent)
+class FileChecksumCache(MultiProcessCache):
+    cache_file_name = "local_file_checksum_cache.dat"
+    CACHE_VERSION = 1
+
+    def __init__(self):
+        self.mtime_cache = FileMtimeCache()
+        MultiProcessCache.__init__(self)
+
+    def get_checksum(self, f):
+        entry = self.cachedata[0].get(f)
+        cmtime = self.mtime_cache.cached_mtime(f)
+        if entry:
+            (mtime, hashval) = entry
+            if cmtime == mtime:
+                return hashval
+            else:
+                bb.debug(2, "file %s changed mtime, recompute checksum" % f)
+
+        hashval = bb.utils.md5_file(f)
+        self.cachedata_extras[0][f] = (cmtime, hashval)
+        return hashval
+
+    def merge_data(self, source, dest):
+        for h in source[0]:
+            if h in dest:
+                (smtime, _) = source[0][h]
+                (dmtime, _) = dest[0][h]
+                if smtime > dmtime:
+                    dest[0][h] = source[0][h]
+            else:
+                dest[0][h] = source[0][h]
+
+    def get_checksums(self, filelist, pn):
+        """Get checksums for a list of files"""
+
+        def checksum_file(f):
+            try:
+                checksum = self.get_checksum(f)
+            except OSError as e:
+                bb.warn("Unable to get checksum for %s SRC_URI entry %s: %s" % (pn, os.path.basename(f), e))
+                return None
+            return checksum
+
+        def checksum_dir(pth):
+            # Handle directories recursively
+            dirchecksums = []
+            for root, dirs, files in os.walk(pth):
+                for name in files:
+                    fullpth = os.path.join(root, name)
+                    checksum = checksum_file(fullpth)
+                    if checksum:
+                        dirchecksums.append((fullpth, checksum))
+            return dirchecksums
+
+        checksums = []
+        for pth in filelist.split():
+            exist = pth.split(":")[1]
+            if exist == "False":
+                continue
+            pth = pth.split(":")[0]
+            if '*' in pth:
+                # Handle globs
+                for f in glob.glob(pth):
+                    if os.path.isdir(f):
+                        if not os.path.islink(f):
+                            checksums.extend(checksum_dir(f))
+                    else:
+                        checksum = checksum_file(f)
+                        checksums.append((f, checksum))
+            elif os.path.isdir(pth):
+                if not os.path.islink(pth):
+                    checksums.extend(checksum_dir(pth))
+            else:
+                checksum = checksum_file(pth)
+                checksums.append((pth, checksum))
+
+        checksums.sort(key=operator.itemgetter(1))
+        return checksums

bitbake/lib/bb/codeparser.py

@@ -0,0 +1,436 @@
+import ast
+import codegen
+import logging
+import os.path
+import bb.utils, bb.data
+from itertools import chain
+from pysh import pyshyacc, pyshlex, sherrors
+from bb.cache import MultiProcessCache
+
+
+logger = logging.getLogger('BitBake.CodeParser')
+
+try:
+    import cPickle as pickle
+except ImportError:
+    import pickle
+    logger.info('Importing cPickle failed.  Falling back to a very slow implementation.')
+
+
+def check_indent(codestr):
+    """If the code is indented, add a top level piece of code to 'remove' the indentation"""
+
+    i = 0
+    while codestr[i] in ["\n", "\t", " "]:
+        i = i + 1
+
+    if i == 0:
+        return codestr
+
+    if codestr[i-1] == "\t" or codestr[i-1] == " ":
+        if codestr[0] == "\n":
+            # Since we're adding a line, we need to remove one line of any empty padding
+            # to ensure line numbers are correct
+            codestr = codestr[1:]
+        return "if 1:\n" + codestr
+
+    return codestr
+
+
+# Basically pickle, in python 2.7.3 at least, does badly with data duplication 
+# upon pickling and unpickling. Combine this with duplicate objects and things
+# are a mess.
+#
+# When the sets are originally created, python calls intern() on the set keys
+# which significantly improves memory usage. Sadly the pickle/unpickle process
+# doesn't call intern() on the keys and results in the same strings being duplicated
+# in memory. This also means pickle will save the same string multiple times in
+# the cache file.
+#
+# By having shell and python cacheline objects with setstate/getstate, we force
+# the object creation through our own routine where we can call intern (via internSet).
+#
+# We also use hashable frozensets and ensure we use references to these so that
+# duplicates can be removed, both in memory and in the resulting pickled data.
+#
+# By playing these games, the size of the cache file shrinks dramatically
+# meaning faster load times and the reloaded cache files also consume much less
+# memory. Smaller cache files, faster load times and lower memory usage is good.
+#
+# A custom getstate/setstate using tuples is actually worth 15% cachesize by
+# avoiding duplication of the attribute names!
+
+class SetCache(object):
+    def __init__(self):
+        self.setcache = {}
+
+    def internSet(self, items):
+        
+        new = []
+        for i in items:
+            new.append(intern(i))
+        s = frozenset(new)
+        if hash(s) in self.setcache:
+            return self.setcache[hash(s)]
+        self.setcache[hash(s)] = s
+        return s
+
+codecache = SetCache()
+
+class pythonCacheLine(object):
+    def __init__(self, refs, execs, contains):
+        self.refs = codecache.internSet(refs)
+        self.execs = codecache.internSet(execs)
+        self.contains = {}
+        for c in contains:
+            self.contains[c] = codecache.internSet(contains[c])
+
+    def __getstate__(self):
+        return (self.refs, self.execs, self.contains)
+
+    def __setstate__(self, state):
+        (refs, execs, contains) = state
+        self.__init__(refs, execs, contains)
+    def __hash__(self):
+        l = (hash(self.refs), hash(self.execs))
+        for c in sorted(self.contains.keys()):
+            l = l + (c, hash(self.contains[c]))
+        return hash(l)
+    def __repr__(self):
+        return " ".join([str(self.refs), str(self.execs), str(self.contains)]) 
+
+
+class shellCacheLine(object):
+    def __init__(self, execs):
+        self.execs = codecache.internSet(execs)
+
+    def __getstate__(self):
+        return (self.execs)
+
+    def __setstate__(self, state):
+        (execs) = state
+        self.__init__(execs)
+    def __hash__(self):
+        return hash(self.execs)
+    def __repr__(self):
+        return str(self.execs)
+
+class CodeParserCache(MultiProcessCache):
+    cache_file_name = "bb_codeparser.dat"
+    CACHE_VERSION = 7
+
+    def __init__(self):
+        MultiProcessCache.__init__(self)
+        self.pythoncache = self.cachedata[0]
+        self.shellcache = self.cachedata[1]
+        self.pythoncacheextras = self.cachedata_extras[0]
+        self.shellcacheextras = self.cachedata_extras[1]
+
+        # To avoid duplication in the codeparser cache, keep
+        # a lookup of hashes of objects we already have
+        self.pythoncachelines = {}
+        self.shellcachelines = {}
+
+    def newPythonCacheLine(self, refs, execs, contains):
+        cacheline = pythonCacheLine(refs, execs, contains)
+        h = hash(cacheline)
+        if h in self.pythoncachelines:
+            return self.pythoncachelines[h]
+        self.pythoncachelines[h] = cacheline
+        return cacheline
+
+    def newShellCacheLine(self, execs):
+        cacheline = shellCacheLine(execs)
+        h = hash(cacheline)
+        if h in self.shellcachelines:
+            return self.shellcachelines[h]
+        self.shellcachelines[h] = cacheline
+        return cacheline
+
+    def init_cache(self, d):
+        # Check if we already have the caches
+        if self.pythoncache:
+            return
+
+        MultiProcessCache.init_cache(self, d)
+
+        # cachedata gets re-assigned in the parent
+        self.pythoncache = self.cachedata[0]
+        self.shellcache = self.cachedata[1]
+
+    def create_cachedata(self):
+        data = [{}, {}]
+        return data
+
+codeparsercache = CodeParserCache()
+
+def parser_cache_init(d):
+    codeparsercache.init_cache(d)
+
+def parser_cache_save():
+    codeparsercache.save_extras()
+
+def parser_cache_savemerge():
+    codeparsercache.save_merge()
+
+Logger = logging.getLoggerClass()
+class BufferedLogger(Logger):
+    def __init__(self, name, level=0, target=None):
+        Logger.__init__(self, name)
+        self.setLevel(level)
+        self.buffer = []
+        self.target = target
+
+    def handle(self, record):
+        self.buffer.append(record)
+
+    def flush(self):
+        for record in self.buffer:
+            self.target.handle(record)
+        self.buffer = []
+
+class PythonParser():
+    getvars = (".getVar", ".appendVar", ".prependVar")
+    containsfuncs = ("bb.utils.contains", "base_contains", "bb.utils.contains_any")
+    execfuncs = ("bb.build.exec_func", "bb.build.exec_task")
+
+    def warn(self, func, arg):
+        """Warn about calls of bitbake APIs which pass a non-literal
+        argument for the variable name, as we're not able to track such
+        a reference.
+        """
+
+        try:
+            funcstr = codegen.to_source(func)
+            argstr = codegen.to_source(arg)
+        except TypeError:
+            self.log.debug(2, 'Failed to convert function and argument to source form')
+        else:
+            self.log.debug(1, self.unhandled_message % (funcstr, argstr))
+
+    def visit_Call(self, node):
+        name = self.called_node_name(node.func)
+        if name and name.endswith(self.getvars) or name in self.containsfuncs:
+            if isinstance(node.args[0], ast.Str):
+                varname = node.args[0].s
+                if name in self.containsfuncs and isinstance(node.args[1], ast.Str):
+                    if varname not in self.contains:
+                        self.contains[varname] = set()
+                    self.contains[varname].add(node.args[1].s)
+                else:                      
+                    self.references.add(node.args[0].s)
+            else:
+                self.warn(node.func, node.args[0])
+        elif name and name.endswith(".expand"):
+            if isinstance(node.args[0], ast.Str):
+                value = node.args[0].s
+                d = bb.data.init()
+                parser = d.expandWithRefs(value, self.name)
+                self.references |= parser.references
+                self.execs |= parser.execs
+                for varname in parser.contains:
+                    if varname not in self.contains:
+                        self.contains[varname] = set()
+                    self.contains[varname] |= parser.contains[varname]
+        elif name in self.execfuncs:
+            if isinstance(node.args[0], ast.Str):
+                self.var_execs.add(node.args[0].s)
+            else:
+                self.warn(node.func, node.args[0])
+        elif name and isinstance(node.func, (ast.Name, ast.Attribute)):
+            self.execs.add(name)
+
+    def called_node_name(self, node):
+        """Given a called node, return its original string form"""
+        components = []
+        while node:
+            if isinstance(node, ast.Attribute):
+                components.append(node.attr)
+                node = node.value
+            elif isinstance(node, ast.Name):
+                components.append(node.id)
+                return '.'.join(reversed(components))
+            else:
+                break
+
+    def __init__(self, name, log):
+        self.name = name
+        self.var_execs = set()
+        self.contains = {}
+        self.execs = set()
+        self.references = set()
+        self.log = BufferedLogger('BitBake.Data.PythonParser', logging.DEBUG, log)
+
+        self.unhandled_message = "in call of %s, argument '%s' is not a string literal"
+        self.unhandled_message = "while parsing %s, %s" % (name, self.unhandled_message)
+
+    def parse_python(self, node, lineno=0, filename="<string>"):
+        if not node or not node.strip():
+            return
+
+        h = hash(str(node))
+
+        if h in codeparsercache.pythoncache:
+            self.references = set(codeparsercache.pythoncache[h].refs)
+            self.execs = set(codeparsercache.pythoncache[h].execs)
+            self.contains = {}
+            for i in codeparsercache.pythoncache[h].contains:
+                self.contains[i] = set(codeparsercache.pythoncache[h].contains[i])
+            return
+
+        if h in codeparsercache.pythoncacheextras:
+            self.references = set(codeparsercache.pythoncacheextras[h].refs)
+            self.execs = set(codeparsercache.pythoncacheextras[h].execs)
+            self.contains = {}
+            for i in codeparsercache.pythoncacheextras[h].contains:
+                self.contains[i] = set(codeparsercache.pythoncacheextras[h].contains[i])
+            return
+
+        # We can't add to the linenumbers for compile, we can pad to the correct number of blank lines though
+        node = "\n" * int(lineno) + node
+        code = compile(check_indent(str(node)), filename, "exec",
+                       ast.PyCF_ONLY_AST)
+
+        for n in ast.walk(code):
+            if n.__class__.__name__ == "Call":
+                self.visit_Call(n)
+
+        self.execs.update(self.var_execs)
+
+        codeparsercache.pythoncacheextras[h] = codeparsercache.newPythonCacheLine(self.references, self.execs, self.contains)
+
+class ShellParser():
+    def __init__(self, name, log):
+        self.funcdefs = set()
+        self.allexecs = set()
+        self.execs = set()
+        self.log = BufferedLogger('BitBake.Data.%s' % name, logging.DEBUG, log)
+        self.unhandled_template = "unable to handle non-literal command '%s'"
+        self.unhandled_template = "while parsing %s, %s" % (name, self.unhandled_template)
+
+    def parse_shell(self, value):
+        """Parse the supplied shell code in a string, returning the external
+        commands it executes.
+        """
+
+        h = hash(str(value))
+
+        if h in codeparsercache.shellcache:
+            self.execs = set(codeparsercache.shellcache[h].execs)
+            return self.execs
+
+        if h in codeparsercache.shellcacheextras:
+            self.execs = set(codeparsercache.shellcacheextras[h].execs)
+            return self.execs
+
+        self._parse_shell(value)
+        self.execs = set(cmd for cmd in self.allexecs if cmd not in self.funcdefs)
+
+        codeparsercache.shellcacheextras[h] = codeparsercache.newShellCacheLine(self.execs)
+
+        return self.execs
+
+    def _parse_shell(self, value):
+        try:
+            tokens, _ = pyshyacc.parse(value, eof=True, debug=False)
+        except pyshlex.NeedMore:
+            raise sherrors.ShellSyntaxError("Unexpected EOF")
+
+        for token in tokens:
+            self.process_tokens(token)
+
+    def process_tokens(self, tokens):
+        """Process a supplied portion of the syntax tree as returned by
+        pyshyacc.parse.
+        """
+
+        def function_definition(value):
+            self.funcdefs.add(value.name)
+            return [value.body], None
+
+        def case_clause(value):
+            # Element 0 of each item in the case is the list of patterns, and
+            # Element 1 of each item in the case is the list of commands to be
+            # executed when that pattern matches.
+            words = chain(*[item[0] for item in value.items])
+            cmds  = chain(*[item[1] for item in value.items])
+            return cmds, words
+
+        def if_clause(value):
+            main = chain(value.cond, value.if_cmds)
+            rest = value.else_cmds
+            if isinstance(rest, tuple) and rest[0] == "elif":
+                return chain(main, if_clause(rest[1]))
+            else:
+                return chain(main, rest)
+
+        def simple_command(value):
+            return None, chain(value.words, (assign[1] for assign in value.assigns))
+
+        token_handlers = {
+            "and_or": lambda x: ((x.left, x.right), None),
+            "async": lambda x: ([x], None),
+            "brace_group": lambda x: (x.cmds, None),
+            "for_clause": lambda x: (x.cmds, x.items),
+            "function_definition": function_definition,
+            "if_clause": lambda x: (if_clause(x), None),
+            "pipeline": lambda x: (x.commands, None),
+            "redirect_list": lambda x: ([x.cmd], None),
+            "subshell": lambda x: (x.cmds, None),
+            "while_clause": lambda x: (chain(x.condition, x.cmds), None),
+            "until_clause": lambda x: (chain(x.condition, x.cmds), None),
+            "simple_command": simple_command,
+            "case_clause": case_clause,
+        }
+
+        for token in tokens:
+            name, value = token
+            try:
+                more_tokens, words = token_handlers[name](value)
+            except KeyError:
+                raise NotImplementedError("Unsupported token type " + name)
+
+            if more_tokens:
+                self.process_tokens(more_tokens)
+
+            if words:
+                self.process_words(words)
+
+    def process_words(self, words):
+        """Process a set of 'words' in pyshyacc parlance, which includes
+        extraction of executed commands from $() blocks, as well as grabbing
+        the command name argument.
+        """
+
+        words = list(words)
+        for word in list(words):
+            wtree = pyshlex.make_wordtree(word[1])
+            for part in wtree:
+                if not isinstance(part, list):
+                    continue
+
+                if part[0] in ('`', '$('):
+                    command = pyshlex.wordtree_as_string(part[1:-1])
+                    self._parse_shell(command)
+
+                    if word[0] in ("cmd_name", "cmd_word"):
+                        if word in words:
+                            words.remove(word)
+
+        usetoken = False
+        for word in words:
+            if word[0] in ("cmd_name", "cmd_word") or \
+               (usetoken and word[0] == "TOKEN"):
+                if "=" in word[1]:
+                    usetoken = True
+                    continue
+
+                cmd = word[1]
+                if cmd.startswith("$"):
+                    self.log.debug(1, self.unhandled_template % cmd)
+                elif cmd == "eval":
+                    command = " ".join(word for _, word in words[1:])
+                    self._parse_shell(command)
+                else:
+                    self.allexecs.add(cmd)
+                break

bitbake/lib/bb/command.py

@@ -0,0 +1,474 @@
+"""
+BitBake 'Command' module
+
+Provide an interface to interact with the bitbake server through 'commands'
+"""
+
+# Copyright (C) 2006-2007  Richard Purdie
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+"""
+The bitbake server takes 'commands' from its UI/commandline.
+Commands are either synchronous or asynchronous.
+Async commands return data to the client in the form of events.
+Sync commands must only return data through the function return value
+and must not trigger events, directly or indirectly.
+Commands are queued in a CommandQueue
+"""
+
+import bb.event
+import bb.cooker
+
+class CommandCompleted(bb.event.Event):
+    pass
+
+class CommandExit(bb.event.Event):
+    def  __init__(self, exitcode):
+        bb.event.Event.__init__(self)
+        self.exitcode = int(exitcode)
+
+class CommandFailed(CommandExit):
+    def __init__(self, message):
+        self.error = message
+        CommandExit.__init__(self, 1)
+
+class CommandError(Exception):
+    pass
+
+class Command:
+    """
+    A queue of asynchronous commands for bitbake
+    """
+    def __init__(self, cooker):
+        self.cooker = cooker
+        self.cmds_sync = CommandsSync()
+        self.cmds_async = CommandsAsync()
+
+        # FIXME Add lock for this
+        self.currentAsyncCommand = None
+
+    def runCommand(self, commandline, ro_only = False):
+        command = commandline.pop(0)
+        if hasattr(CommandsSync, command):
+            # Can run synchronous commands straight away
+            command_method = getattr(self.cmds_sync, command)
+            if ro_only:
+                if not hasattr(command_method, 'readonly') or False == getattr(command_method, 'readonly'):
+                    return None, "Not able to execute not readonly commands in readonly mode"
+            try:
+                if getattr(command_method, 'needconfig', False):
+                    self.cooker.updateCacheSync()
+                result = command_method(self, commandline)
+            except CommandError as exc:
+                return None, exc.args[0]
+            except (Exception, SystemExit):
+                import traceback
+                return None, traceback.format_exc()
+            else:
+                return result, None
+        if self.currentAsyncCommand is not None:
+            return None, "Busy (%s in progress)" % self.currentAsyncCommand[0]
+        if command not in CommandsAsync.__dict__:
+            return None, "No such command"
+        self.currentAsyncCommand = (command, commandline)
+        self.cooker.configuration.server_register_idlecallback(self.cooker.runCommands, self.cooker)
+        return True, None
+
+    def runAsyncCommand(self):
+        try:
+            if self.cooker.state in (bb.cooker.state.error, bb.cooker.state.shutdown, bb.cooker.state.forceshutdown):
+                # updateCache will trigger a shutdown of the parser
+                # and then raise BBHandledException triggering an exit
+                self.cooker.updateCache()
+                return False
+            if self.currentAsyncCommand is not None:
+                (command, options) = self.currentAsyncCommand
+                commandmethod = getattr(CommandsAsync, command)
+                needcache = getattr( commandmethod, "needcache" )
+                if needcache and self.cooker.state != bb.cooker.state.running:
+                    self.cooker.updateCache()
+                    return True
+                else:
+                    commandmethod(self.cmds_async, self, options)
+                    return False
+            else:
+                return False
+        except KeyboardInterrupt as exc:
+            self.finishAsyncCommand("Interrupted")
+            return False
+        except SystemExit as exc:
+            arg = exc.args[0]
+            if isinstance(arg, basestring):
+                self.finishAsyncCommand(arg)
+            else:
+                self.finishAsyncCommand("Exited with %s" % arg)
+            return False
+        except Exception as exc:
+            import traceback
+            if isinstance(exc, bb.BBHandledException):
+                self.finishAsyncCommand("")
+            else:
+                self.finishAsyncCommand(traceback.format_exc())
+            return False
+
+    def finishAsyncCommand(self, msg=None, code=None):
+        if msg or msg == "":
+            bb.event.fire(CommandFailed(msg), self.cooker.expanded_data)
+        elif code:
+            bb.event.fire(CommandExit(code), self.cooker.expanded_data)
+        else:
+            bb.event.fire(CommandCompleted(), self.cooker.expanded_data)
+        self.currentAsyncCommand = None
+        self.cooker.finishcommand()
+
+class CommandsSync:
+    """
+    A class of synchronous commands
+    These should run quickly so as not to hurt interactive performance.
+    These must not influence any running synchronous command.
+    """
+
+    def stateShutdown(self, command, params):
+        """
+        Trigger cooker 'shutdown' mode
+        """
+        command.cooker.shutdown(False)
+
+    def stateForceShutdown(self, command, params):
+        """
+        Stop the cooker
+        """
+        command.cooker.shutdown(True)
+
+    def getAllKeysWithFlags(self, command, params):
+        """
+        Returns a dump of the global state. Call with
+        variable flags to be retrieved as params.
+        """
+        flaglist = params[0]
+        return command.cooker.getAllKeysWithFlags(flaglist)
+    getAllKeysWithFlags.readonly = True
+
+    def getVariable(self, command, params):
+        """
+        Read the value of a variable from data
+        """
+        varname = params[0]
+        expand = True
+        if len(params) > 1:
+            expand = (params[1] == "True")
+
+        return command.cooker.data.getVar(varname, expand)
+    getVariable.readonly = True
+
+    def setVariable(self, command, params):
+        """
+        Set the value of variable in data
+        """
+        varname = params[0]
+        value = str(params[1])
+        command.cooker.data.setVar(varname, value)
+
+    def getSetVariable(self, command, params):
+        """
+        Read the value of a variable from data and set it into the datastore
+        which effectively expands and locks the value.
+        """
+        varname = params[0]
+        result = self.getVariable(command, params)
+        command.cooker.data.setVar(varname, result)
+        return result
+
+    def setConfig(self, command, params):
+        """
+        Set the value of variable in configuration
+        """
+        varname = params[0]
+        value = str(params[1])
+        setattr(command.cooker.configuration, varname, value)
+
+    def enableDataTracking(self, command, params):
+        """
+        Enable history tracking for variables
+        """
+        command.cooker.enableDataTracking()
+
+    def disableDataTracking(self, command, params):
+        """
+        Disable history tracking for variables
+        """
+        command.cooker.disableDataTracking()
+
+    def setPrePostConfFiles(self, command, params):
+        prefiles = params[0].split()
+        postfiles = params[1].split()
+        command.cooker.configuration.prefile = prefiles
+        command.cooker.configuration.postfile = postfiles
+    setPrePostConfFiles.needconfig = False
+
+    def getCpuCount(self, command, params):
+        """
+        Get the CPU count on the bitbake server
+        """
+        return bb.utils.cpu_count()
+    getCpuCount.readonly = True
+    getCpuCount.needconfig = False
+
+    def matchFile(self, command, params):
+        fMatch = params[0]
+        return command.cooker.matchFile(fMatch)
+    matchFile.needconfig = False
+
+    def generateNewImage(self, command, params):
+        image = params[0]
+        base_image = params[1]
+        package_queue = params[2]
+        timestamp = params[3]
+        description = params[4]
+        return command.cooker.generateNewImage(image, base_image,
+                        package_queue, timestamp, description)
+
+    def ensureDir(self, command, params):
+        directory = params[0]
+        bb.utils.mkdirhier(directory)
+    ensureDir.needconfig = False
+
+    def setVarFile(self, command, params):
+        """
+        Save a variable in a file; used for saving in a configuration file
+        """
+        var = params[0]
+        val = params[1]
+        default_file = params[2]
+        op = params[3]
+        command.cooker.modifyConfigurationVar(var, val, default_file, op)
+    setVarFile.needconfig = False
+
+    def removeVarFile(self, command, params):
+        """
+        Remove a variable declaration from a file
+        """
+        var = params[0]
+        command.cooker.removeConfigurationVar(var)
+    removeVarFile.needconfig = False
+
+    def createConfigFile(self, command, params):
+        """
+        Create an extra configuration file
+        """
+        name = params[0]
+        command.cooker.createConfigFile(name)
+    createConfigFile.needconfig = False
+
+    def setEventMask(self, command, params):
+        handlerNum = params[0]
+        llevel = params[1]
+        debug_domains = params[2]
+        mask = params[3]
+        return bb.event.set_UIHmask(handlerNum, llevel, debug_domains, mask)
+    setEventMask.needconfig = False
+    setEventMask.readonly = True
+
+    def setFeatures(self, command, params):
+        """
+        Set the cooker features to include the passed list of features
+        """
+        features = params[0]
+        command.cooker.setFeatures(features)
+    setFeatures.needconfig = False
+    # although we change the internal state of the cooker, this is transparent since
+    # we always take and leave the cooker in state.initial
+    setFeatures.readonly = True
+
+    def updateConfig(self, command, params):
+        options = params[0]
+        environment = params[1]
+        command.cooker.updateConfigOpts(options, environment)
+    updateConfig.needconfig = False
+
+class CommandsAsync:
+    """
+    A class of asynchronous commands
+    These functions communicate via generated events.
+    Any function that requires metadata parsing should be here.
+    """
+
+    def buildFile(self, command, params):
+        """
+        Build a single specified .bb file
+        """
+        bfile = params[0]
+        task = params[1]
+
+        command.cooker.buildFile(bfile, task)
+    buildFile.needcache = False
+
+    def buildTargets(self, command, params):
+        """
+        Build a set of targets
+        """
+        pkgs_to_build = params[0]
+        task = params[1]
+
+        command.cooker.buildTargets(pkgs_to_build, task)
+    buildTargets.needcache = True
+
+    def generateDepTreeEvent(self, command, params):
+        """
+        Generate an event containing the dependency information
+        """
+        pkgs_to_build = params[0]
+        task = params[1]
+
+        command.cooker.generateDepTreeEvent(pkgs_to_build, task)
+        command.finishAsyncCommand()
+    generateDepTreeEvent.needcache = True
+
+    def generateDotGraph(self, command, params):
+        """
+        Dump dependency information to disk as .dot files
+        """
+        pkgs_to_build = params[0]
+        task = params[1]
+
+        command.cooker.generateDotGraphFiles(pkgs_to_build, task)
+        command.finishAsyncCommand()
+    generateDotGraph.needcache = True
+
+    def generateTargetsTree(self, command, params):
+        """
+        Generate a tree of buildable targets.
+        If klass is provided ensure all recipes that inherit the class are
+        included in the package list.
+        If pkg_list provided use that list (plus any extras brought in by
+        klass) rather than generating a tree for all packages.
+        """
+        klass = params[0]
+        pkg_list = params[1]
+
+        command.cooker.generateTargetsTree(klass, pkg_list)
+        command.finishAsyncCommand()
+    generateTargetsTree.needcache = True
+
+    def findCoreBaseFiles(self, command, params):
+        """
+        Find certain files in COREBASE directory. i.e. Layers
+        """
+        subdir = params[0]
+        filename = params[1]
+
+        command.cooker.findCoreBaseFiles(subdir, filename)
+        command.finishAsyncCommand()
+    findCoreBaseFiles.needcache = False
+
+    def findConfigFiles(self, command, params):
+        """
+        Find config files which provide appropriate values
+        for the passed configuration variable. i.e. MACHINE
+        """
+        varname = params[0]
+
+        command.cooker.findConfigFiles(varname)
+        command.finishAsyncCommand()
+    findConfigFiles.needcache = False
+
+    def findFilesMatchingInDir(self, command, params):
+        """
+        Find implementation files matching the specified pattern
+        in the requested subdirectory of a BBPATH
+        """
+        pattern = params[0]
+        directory = params[1]
+
+        command.cooker.findFilesMatchingInDir(pattern, directory)
+        command.finishAsyncCommand()
+    findFilesMatchingInDir.needcache = False
+
+    def findConfigFilePath(self, command, params):
+        """
+        Find the path of the requested configuration file
+        """
+        configfile = params[0]
+
+        command.cooker.findConfigFilePath(configfile)
+        command.finishAsyncCommand()
+    findConfigFilePath.needcache = False
+
+    def showVersions(self, command, params):
+        """
+        Show the currently selected versions
+        """
+        command.cooker.showVersions()
+        command.finishAsyncCommand()
+    showVersions.needcache = True
+
+    def showEnvironmentTarget(self, command, params):
+        """
+        Print the environment of a target recipe
+        (needs the cache to work out which recipe to use)
+        """
+        pkg = params[0]
+
+        command.cooker.showEnvironment(None, pkg)
+        command.finishAsyncCommand()
+    showEnvironmentTarget.needcache = True
+
+    def showEnvironment(self, command, params):
+        """
+        Print the standard environment
+        or if specified the environment for a specified recipe
+        """
+        bfile = params[0]
+
+        command.cooker.showEnvironment(bfile)
+        command.finishAsyncCommand()
+    showEnvironment.needcache = False
+
+    def parseFiles(self, command, params):
+        """
+        Parse the .bb files
+        """
+        command.cooker.updateCache()
+        command.finishAsyncCommand()
+    parseFiles.needcache = True
+
+    def compareRevisions(self, command, params):
+        """
+        Parse the .bb files
+        """
+        if bb.fetch.fetcher_compare_revisions(command.cooker.data):
+            command.finishAsyncCommand(code=1)
+        else:
+            command.finishAsyncCommand()
+    compareRevisions.needcache = True
+
+    def triggerEvent(self, command, params):
+        """
+        Trigger a certain event
+        """
+        event = params[0]
+        bb.event.fire(eval(event), command.cooker.data)
+        command.currentAsyncCommand = None
+    triggerEvent.needcache = False
+
+    def resetCooker(self, command, params):
+        """
+        Reset the cooker to its initial state, thus forcing a reparse for
+        any async command that has the needcache property set to True
+        """
+        command.cooker.reset()
+        command.finishAsyncCommand()
+    resetCooker.needcache = False
+

bitbake/lib/bb/compat.py

@@ -0,0 +1,6 @@
+"""Code pulled from future python versions, here for compatibility"""
+
+from collections import MutableMapping, KeysView, ValuesView, ItemsView, OrderedDict
+from functools import total_ordering
+
+

bitbake/lib/bb/cookerdata.py

@@ -0,0 +1,345 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (C) 2003, 2004  Chris Larson
+# Copyright (C) 2003, 2004  Phil Blundell
+# Copyright (C) 2003 - 2005 Michael 'Mickey' Lauer
+# Copyright (C) 2005        Holger Hans Peter Freyther
+# Copyright (C) 2005        ROAD GmbH
+# Copyright (C) 2006        Richard Purdie
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os, sys
+from functools import wraps
+import logging
+import bb
+from bb import data
+import bb.parse
+
+logger      = logging.getLogger("BitBake")
+parselog    = logging.getLogger("BitBake.Parsing")
+
+class ConfigParameters(object):
+    def __init__(self, argv=sys.argv):
+        self.options, targets = self.parseCommandLine(argv)
+        self.environment = self.parseEnvironment()
+
+        self.options.pkgs_to_build = targets or []
+
+        self.options.tracking = False
+        if hasattr(self.options, "show_environment") and self.options.show_environment:
+            self.options.tracking = True
+
+        for key, val in self.options.__dict__.items():
+            setattr(self, key, val)
+
+    def parseCommandLine(self, argv=sys.argv):
+        raise Exception("Caller must implement commandline option parsing")
+
+    def parseEnvironment(self):
+        return os.environ.copy()
+
+    def updateFromServer(self, server):
+        if not self.options.cmd:
+            defaulttask, error = server.runCommand(["getVariable", "BB_DEFAULT_TASK"])
+            if error:
+                raise Exception("Unable to get the value of BB_DEFAULT_TASK from the server: %s" % error)
+            self.options.cmd = defaulttask or "build"
+        _, error = server.runCommand(["setConfig", "cmd", self.options.cmd])
+        if error:
+            raise Exception("Unable to set configuration option 'cmd' on the server: %s" % error)
+
+        if not self.options.pkgs_to_build:
+            bbpkgs, error = server.runCommand(["getVariable", "BBTARGETS"])
+            if error:
+                raise Exception("Unable to get the value of BBTARGETS from the server: %s" % error)
+            if bbpkgs:
+                self.options.pkgs_to_build.extend(bbpkgs.split())
+
+    def updateToServer(self, server, environment):
+        options = {}
+        for o in ["abort", "tryaltconfigs", "force", "invalidate_stamp", 
+                  "verbose", "debug", "dry_run", "dump_signatures", 
+                  "debug_domains", "extra_assume_provided", "profile",
+                  "prefile", "postfile"]:
+            options[o] = getattr(self.options, o)
+
+        ret, error = server.runCommand(["updateConfig", options, environment])
+        if error:
+                raise Exception("Unable to update the server configuration with local parameters: %s" % error)
+
+    def parseActions(self):
+        # Parse any commandline into actions
+        action = {'action':None, 'msg':None}
+        if self.options.show_environment:
+            if 'world' in self.options.pkgs_to_build:
+                action['msg'] = "'world' is not a valid target for --environment."
+            elif 'universe' in self.options.pkgs_to_build:
+                action['msg'] = "'universe' is not a valid target for --environment."
+            elif len(self.options.pkgs_to_build) > 1:
+                action['msg'] = "Only one target can be used with the --environment option."
+            elif self.options.buildfile and len(self.options.pkgs_to_build) > 0:
+                action['msg'] = "No target should be used with the --environment and --buildfile options."
+            elif len(self.options.pkgs_to_build) > 0:
+                action['action'] = ["showEnvironmentTarget", self.options.pkgs_to_build]
+            else:
+                action['action'] = ["showEnvironment", self.options.buildfile]
+        elif self.options.buildfile is not None:
+            action['action'] = ["buildFile", self.options.buildfile, self.options.cmd]
+        elif self.options.revisions_changed:
+            action['action'] = ["compareRevisions"]
+        elif self.options.show_versions:
+            action['action'] = ["showVersions"]
+        elif self.options.parse_only:
+            action['action'] = ["parseFiles"]
+        elif self.options.dot_graph:
+            if self.options.pkgs_to_build:
+                action['action'] = ["generateDotGraph", self.options.pkgs_to_build, self.options.cmd]
+            else:
+                action['msg'] = "Please specify a package name for dependency graph generation."
+        else:
+            if self.options.pkgs_to_build:
+                action['action'] = ["buildTargets", self.options.pkgs_to_build, self.options.cmd]
+            else:
+                #action['msg'] = "Nothing to do.  Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information."
+                action = None
+        self.options.initialaction = action
+        return action
+
+class CookerConfiguration(object):
+    """
+    Manages build options and configurations for one run
+    """
+
+    def __init__(self):
+        self.debug_domains = []
+        self.extra_assume_provided = []
+        self.prefile = []
+        self.postfile = []
+        self.prefile_server = []
+        self.postfile_server = []
+        self.debug = 0
+        self.cmd = None
+        self.abort = True
+        self.force = False
+        self.profile = False
+        self.nosetscene = False
+        self.setsceneonly = False
+        self.invalidate_stamp = False
+        self.dump_signatures = []
+        self.dry_run = False
+        self.tracking = False
+        self.interface = []
+        self.writeeventlog = False
+
+        self.env = {}
+
+    def setConfigParameters(self, parameters):
+        for key in self.__dict__.keys():
+            if key in parameters.options.__dict__:
+                setattr(self, key, parameters.options.__dict__[key])
+        self.env = parameters.environment.copy()
+        self.tracking = parameters.tracking
+
+    def setServerRegIdleCallback(self, srcb):
+        self.server_register_idlecallback = srcb
+
+    def __getstate__(self):
+        state = {}
+        for key in self.__dict__.keys():
+            if key == "server_register_idlecallback":
+                state[key] = None
+            else:
+                state[key] = getattr(self, key)
+        return state
+
+    def __setstate__(self,state):
+        for k in state:
+            setattr(self, k, state[k]) 
+
+
+def catch_parse_error(func):
+    """Exception handling bits for our parsing"""
+    @wraps(func)
+    def wrapped(fn, *args):
+        try:
+            return func(fn, *args)
+        except IOError as exc:
+            import traceback
+            parselog.critical(traceback.format_exc())
+            parselog.critical("Unable to parse %s: %s" % (fn, exc))
+            sys.exit(1)
+        except bb.data_smart.ExpansionError as exc:
+            import traceback
+
+            bbdir = os.path.dirname(__file__) + os.sep
+            exc_class, exc, tb = sys.exc_info()
+            for tb in iter(lambda: tb.tb_next, None):
+                # Skip frames in bitbake itself, we only want the metadata
+                fn, _, _, _ = traceback.extract_tb(tb, 1)[0]
+                if not fn.startswith(bbdir):
+                    break
+            parselog.critical("Unable to parse %s", fn, exc_info=(exc_class, exc, tb))
+        except bb.parse.ParseError as exc:
+            parselog.critical(str(exc))
+            sys.exit(1)
+    return wrapped
+
+@catch_parse_error
+def parse_config_file(fn, data, include=True):
+    return bb.parse.handle(fn, data, include)
+
+@catch_parse_error
+def _inherit(bbclass, data):
+    bb.parse.BBHandler.inherit(bbclass, "configuration INHERITs", 0, data)
+    return data
+
+def findConfigFile(configfile, data):
+    search = []
+    bbpath = data.getVar("BBPATH", True)
+    if bbpath:
+        for i in bbpath.split(":"):
+            search.append(os.path.join(i, "conf", configfile))
+    path = os.getcwd()
+    while path != "/":
+        search.append(os.path.join(path, "conf", configfile))
+        path, _ = os.path.split(path)
+
+    for i in search:
+        if os.path.exists(i):
+            return i
+
+    return None
+
+class CookerDataBuilder(object):
+
+    def __init__(self, cookercfg, worker = False):
+
+        self.prefiles = cookercfg.prefile
+        self.postfiles = cookercfg.postfile
+        self.tracking = cookercfg.tracking
+
+        bb.utils.set_context(bb.utils.clean_context())
+        bb.event.set_class_handlers(bb.event.clean_class_handlers())
+        self.data = bb.data.init()
+        if self.tracking:
+            self.data.enableTracking()
+
+        # Keep a datastore of the initial environment variables and their
+        # values from when BitBake was launched to enable child processes
+        # to use environment variables which have been cleaned from the
+        # BitBake processes env
+        self.savedenv = bb.data.init()
+        for k in cookercfg.env:
+            self.savedenv.setVar(k, cookercfg.env[k])
+
+        filtered_keys = bb.utils.approved_variables()
+        bb.data.inheritFromOS(self.data, self.savedenv, filtered_keys)
+        self.data.setVar("BB_ORIGENV", self.savedenv)
+        
+        if worker:
+            self.data.setVar("BB_WORKERCONTEXT", "1")
+
+    def parseBaseConfiguration(self):
+        try:
+            self.parseConfigurationFiles(self.prefiles, self.postfiles)
+        except SyntaxError:
+            raise bb.BBHandledException
+        except bb.data_smart.ExpansionError as e:
+            logger.error(str(e))
+            raise bb.BBHandledException
+        except Exception:
+            logger.exception("Error parsing configuration files")
+            raise bb.BBHandledException
+
+    def _findLayerConf(self, data):
+        return findConfigFile("bblayers.conf", data)
+
+    def parseConfigurationFiles(self, prefiles, postfiles):
+        data = self.data
+        bb.parse.init_parser(data)
+
+        # Parse files for loading *before* bitbake.conf and any includes
+        for f in prefiles:
+            data = parse_config_file(f, data)
+
+        layerconf = self._findLayerConf(data)
+        if layerconf:
+            parselog.debug(2, "Found bblayers.conf (%s)", layerconf)
+            # By definition bblayers.conf is in conf/ of TOPDIR.
+            # We may have been called with cwd somewhere else so reset TOPDIR
+            data.setVar("TOPDIR", os.path.dirname(os.path.dirname(layerconf)))
+            data = parse_config_file(layerconf, data)
+
+            layers = (data.getVar('BBLAYERS', True) or "").split()
+
+            data = bb.data.createCopy(data)
+            approved = bb.utils.approved_variables()
+            for layer in layers:
+                parselog.debug(2, "Adding layer %s", layer)
+                if 'HOME' in approved and '~' in layer:
+                    layer = os.path.expanduser(layer)
+                if layer.endswith('/'):
+                    layer = layer.rstrip('/')
+                data.setVar('LAYERDIR', layer)
+                data = parse_config_file(os.path.join(layer, "conf", "layer.conf"), data)
+                data.expandVarref('LAYERDIR')
+
+            data.delVar('LAYERDIR')
+
+        if not data.getVar("BBPATH", True):
+            msg = "The BBPATH variable is not set"
+            if not layerconf:
+                msg += (" and bitbake did not find a conf/bblayers.conf file in"
+                        " the expected location.\nMaybe you accidentally"
+                        " invoked bitbake from the wrong directory?")
+            raise SystemExit(msg)
+
+        data = parse_config_file(os.path.join("conf", "bitbake.conf"), data)
+
+        # Parse files for loading *after* bitbake.conf and any includes
+        for p in postfiles:
+            data = parse_config_file(p, data)
+
+        # Handle any INHERITs and inherit the base class
+        bbclasses  = ["base"] + (data.getVar('INHERIT', True) or "").split()
+        for bbclass in bbclasses:
+            data = _inherit(bbclass, data)
+
+        # Nomally we only register event handlers at the end of parsing .bb files
+        # We register any handlers we've found so far here...
+        for var in data.getVar('__BBHANDLERS', False) or []:
+            handlerfn = data.getVarFlag(var, "filename", False)
+            handlerln = int(data.getVarFlag(var, "lineno", False))
+            bb.event.register(var, data.getVar(var, False),  (data.getVarFlag(var, "eventmask", True) or "").split(), handlerfn, handlerln)
+
+        if data.getVar("BB_WORKERCONTEXT", False) is None:
+            bb.fetch.fetcher_init(data)
+        bb.codeparser.parser_cache_init(data)
+        bb.event.fire(bb.event.ConfigParsed(), data)
+
+        if data.getVar("BB_INVALIDCONF", False) is True:
+            data.setVar("BB_INVALIDCONF", False)
+            self.parseConfigurationFiles(self.prefiles, self.postfiles)
+            return
+
+        bb.parse.init_parser(data)
+        data.setVar('BBINCLUDED',bb.parse.get_file_depends(data))
+        self.data = data
+        self.data_hash = data.get_hash()
+
+
+

bitbake/lib/bb/daemonize.py

@@ -0,0 +1,193 @@
+"""
+Python Daemonizing helper
+
+Configurable daemon behaviors:
+
+    1.) The current working directory set to the "/" directory.
+    2.) The current file creation mode mask set to 0.
+    3.) Close all open files (1024). 
+    4.) Redirect standard I/O streams to "/dev/null".
+
+A failed call to fork() now raises an exception.
+
+References:
+    1) Advanced Programming in the Unix Environment: W. Richard Stevens
+	http://www.apuebook.com/apue3e.html
+    2) The Linux Programming Interface: Michael Kerrisk
+	http://man7.org/tlpi/index.html
+    3) Unix Programming Frequently Asked Questions:
+	http://www.faqs.org/faqs/unix-faq/programmer/faq/
+
+Modified to allow a function to be daemonized and return for 
+bitbake use by Richard Purdie
+"""
+
+__author__ = "Chad J. Schroeder"
+__copyright__ = "Copyright (C) 2005 Chad J. Schroeder"
+__version__ = "0.2"
+
+# Standard Python modules.
+import os                    # Miscellaneous OS interfaces.
+import sys                   # System-specific parameters and functions.
+
+# Default daemon parameters.
+# File mode creation mask of the daemon.
+# For BitBake's children, we do want to inherit the parent umask.
+UMASK = None
+
+# Default maximum for the number of available file descriptors.
+MAXFD = 1024
+
+# The standard I/O file descriptors are redirected to /dev/null by default.
+if (hasattr(os, "devnull")):
+    REDIRECT_TO = os.devnull
+else:
+    REDIRECT_TO = "/dev/null"
+
+def createDaemon(function, logfile):
+    """
+    Detach a process from the controlling terminal and run it in the
+    background as a daemon, returning control to the caller.
+    """
+
+    try:
+        # Fork a child process so the parent can exit.  This returns control to
+        # the command-line or shell.  It also guarantees that the child will not
+        # be a process group leader, since the child receives a new process ID
+        # and inherits the parent's process group ID.  This step is required
+        # to insure that the next call to os.setsid is successful.
+        pid = os.fork()
+    except OSError as e:
+        raise Exception("%s [%d]" % (e.strerror, e.errno))
+
+    if (pid == 0):      # The first child.
+        # To become the session leader of this new session and the process group
+        # leader of the new process group, we call os.setsid().  The process is
+        # also guaranteed not to have a controlling terminal.
+        os.setsid()
+
+        # Is ignoring SIGHUP necessary?
+        #
+        # It's often suggested that the SIGHUP signal should be ignored before
+        # the second fork to avoid premature termination of the process.  The
+        # reason is that when the first child terminates, all processes, e.g.
+        # the second child, in the orphaned group will be sent a SIGHUP.
+        #
+        # "However, as part of the session management system, there are exactly
+        # two cases where SIGHUP is sent on the death of a process:
+        #
+        #    1) When the process that dies is the session leader of a session that
+        #        is attached to a terminal device, SIGHUP is sent to all processes
+        #        in the foreground process group of that terminal device.
+        #    2) When the death of a process causes a process group to become
+        #        orphaned, and one or more processes in the orphaned group are
+        #        stopped, then SIGHUP and SIGCONT are sent to all members of the
+        #        orphaned group." [2]
+        #
+        # The first case can be ignored since the child is guaranteed not to have
+        # a controlling terminal.  The second case isn't so easy to dismiss.
+        # The process group is orphaned when the first child terminates and
+        # POSIX.1 requires that every STOPPED process in an orphaned process
+        # group be sent a SIGHUP signal followed by a SIGCONT signal.  Since the
+        # second child is not STOPPED though, we can safely forego ignoring the
+        # SIGHUP signal.  In any case, there are no ill-effects if it is ignored.
+        #
+        # import signal              # Set handlers for asynchronous events.
+        # signal.signal(signal.SIGHUP, signal.SIG_IGN)
+
+        try:
+            # Fork a second child and exit immediately to prevent zombies.  This
+            # causes the second child process to be orphaned, making the init
+            # process responsible for its cleanup.  And, since the first child is
+            # a session leader without a controlling terminal, it's possible for
+            # it to acquire one by opening a terminal in the future (System V-
+            # based systems).  This second fork guarantees that the child is no
+            # longer a session leader, preventing the daemon from ever acquiring
+            # a controlling terminal.
+            pid = os.fork()     # Fork a second child.
+        except OSError as e:
+            raise Exception("%s [%d]" % (e.strerror, e.errno))
+
+        if (pid == 0):  # The second child.
+            # We probably don't want the file mode creation mask inherited from
+            # the parent, so we give the child complete control over permissions.
+            if UMASK is not None:
+                os.umask(UMASK)
+        else:
+            # Parent (the first child) of the second child.
+            os._exit(0)
+    else:
+        # exit() or _exit()?
+        # _exit is like exit(), but it doesn't call any functions registered
+        # with atexit (and on_exit) or any registered signal handlers.  It also
+        # closes any open file descriptors.  Using exit() may cause all stdio
+        # streams to be flushed twice and any temporary files may be unexpectedly
+        # removed.  It's therefore recommended that child branches of a fork()
+        # and the parent branch(es) of a daemon use _exit().
+        return
+
+    # Close all open file descriptors.  This prevents the child from keeping
+    # open any file descriptors inherited from the parent.  There is a variety
+    # of methods to accomplish this task.  Three are listed below.
+    #
+    # Try the system configuration variable, SC_OPEN_MAX, to obtain the maximum
+    # number of open file descriptors to close.  If it doesn't exist, use
+    # the default value (configurable).
+    #
+    # try:
+    #     maxfd = os.sysconf("SC_OPEN_MAX")
+    # except (AttributeError, ValueError):
+    #     maxfd = MAXFD
+    #
+    # OR
+    #
+    # if (os.sysconf_names.has_key("SC_OPEN_MAX")):
+    #     maxfd = os.sysconf("SC_OPEN_MAX")
+    # else:
+    #     maxfd = MAXFD
+    #
+    # OR
+    #
+    # Use the getrlimit method to retrieve the maximum file descriptor number
+    # that can be opened by this process.  If there is no limit on the
+    # resource, use the default value.
+    #
+    import resource             # Resource usage information.
+    maxfd = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
+    if (maxfd == resource.RLIM_INFINITY):
+        maxfd = MAXFD
+  
+    # Iterate through and close all file descriptors.
+#    for fd in range(0, maxfd):
+#        try:
+#            os.close(fd)
+#        except OSError:        # ERROR, fd wasn't open to begin with (ignored)
+#            pass
+
+    # Redirect the standard I/O file descriptors to the specified file.  Since
+    # the daemon has no controlling terminal, most daemons redirect stdin,
+    # stdout, and stderr to /dev/null.  This is done to prevent side-effects
+    # from reads and writes to the standard I/O file descriptors.
+
+    # This call to open is guaranteed to return the lowest file descriptor,
+    # which will be 0 (stdin), since it was closed above.
+#    os.open(REDIRECT_TO, os.O_RDWR)    # standard input (0)
+
+    # Duplicate standard input to standard output and standard error.
+#    os.dup2(0, 1)                      # standard output (1)
+#    os.dup2(0, 2)                      # standard error (2)
+
+
+    si = file('/dev/null', 'r')
+    so = file(logfile, 'w')
+    se = so
+
+
+    # Replace those fds with our own
+    os.dup2(si.fileno(), sys.stdin.fileno())
+    os.dup2(so.fileno(), sys.stdout.fileno())
+    os.dup2(se.fileno(), sys.stderr.fileno())
+
+    function()
+
+    os._exit(0)

bitbake/lib/bb/data.py

@@ -0,0 +1,448 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Data' implementations
+
+Functions for interacting with the data structure used by the
+BitBake build tools.
+
+The expandKeys and update_data are the most expensive
+operations. At night the cookie monster came by and
+suggested 'give me cookies on setting the variables and
+things will work out'. Taking this suggestion into account
+applying the skills from the not yet passed 'Entwurf und
+Analyse von Algorithmen' lecture and the cookie
+monster seems to be right. We will track setVar more carefully
+to have faster update_data and expandKeys operations.
+
+This is a trade-off between speed and memory again but
+the speed is more critical here.
+"""
+
+# Copyright (C) 2003, 2004  Chris Larson
+# Copyright (C) 2005        Holger Hans Peter Freyther
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+import sys, os, re
+if sys.argv[0][-5:] == "pydoc":
+    path = os.path.dirname(os.path.dirname(sys.argv[1]))
+else:
+    path = os.path.dirname(os.path.dirname(sys.argv[0]))
+sys.path.insert(0, path)
+from itertools import groupby
+
+from bb import data_smart
+from bb import codeparser
+import bb
+
+logger = data_smart.logger
+_dict_type = data_smart.DataSmart
+
+def init():
+    """Return a new object representing the Bitbake data"""
+    return _dict_type()
+
+def init_db(parent = None):
+    """Return a new object representing the Bitbake data,
+    optionally based on an existing object"""
+    if parent is not None:
+        return parent.createCopy()
+    else:
+        return _dict_type()
+
+def createCopy(source):
+    """Link the source set to the destination
+    If one does not find the value in the destination set,
+    search will go on to the source set to get the value.
+    Value from source are copy-on-write. i.e. any try to
+    modify one of them will end up putting the modified value
+    in the destination set.
+    """
+    return source.createCopy()
+
+def initVar(var, d):
+    """Non-destructive var init for data structure"""
+    d.initVar(var)
+
+
+def setVar(var, value, d):
+    """Set a variable to a given value"""
+    d.setVar(var, value)
+
+
+def getVar(var, d, exp = False):
+    """Gets the value of a variable"""
+    return d.getVar(var, exp)
+
+
+def renameVar(key, newkey, d):
+    """Renames a variable from key to newkey"""
+    d.renameVar(key, newkey)
+
+def delVar(var, d):
+    """Removes a variable from the data set"""
+    d.delVar(var)
+
+def appendVar(var, value, d):
+    """Append additional value to a variable"""
+    d.appendVar(var, value)
+
+def setVarFlag(var, flag, flagvalue, d):
+    """Set a flag for a given variable to a given value"""
+    d.setVarFlag(var, flag, flagvalue)
+
+def getVarFlag(var, flag, d):
+    """Gets given flag from given var"""
+    return d.getVarFlag(var, flag, False)
+
+def delVarFlag(var, flag, d):
+    """Removes a given flag from the variable's flags"""
+    d.delVarFlag(var, flag)
+
+def setVarFlags(var, flags, d):
+    """Set the flags for a given variable
+
+    Note:
+        setVarFlags will not clear previous
+        flags. Think of this method as
+        addVarFlags
+    """
+    d.setVarFlags(var, flags)
+
+def getVarFlags(var, d):
+    """Gets a variable's flags"""
+    return d.getVarFlags(var)
+
+def delVarFlags(var, d):
+    """Removes a variable's flags"""
+    d.delVarFlags(var)
+
+def keys(d):
+    """Return a list of keys in d"""
+    return d.keys()
+
+
+__expand_var_regexp__ = re.compile(r"\${[^{}]+}")
+__expand_python_regexp__ = re.compile(r"\${@.+?}")
+
+def expand(s, d, varname = None):
+    """Variable expansion using the data store"""
+    return d.expand(s, varname)
+
+def expandKeys(alterdata, readdata = None):
+    if readdata == None:
+        readdata = alterdata
+
+    todolist = {}
+    for key in alterdata:
+        if not '${' in key:
+            continue
+
+        ekey = expand(key, readdata)
+        if key == ekey:
+            continue
+        todolist[key] = ekey
+
+    # These two for loops are split for performance to maximise the
+    # usefulness of the expand cache
+    for key in sorted(todolist):
+        ekey = todolist[key]
+        newval = alterdata.getVar(ekey, False)
+        if newval is not None:
+            val = alterdata.getVar(key, False)
+            if val is not None:
+                bb.warn("Variable key %s (%s) replaces original key %s (%s)." % (key, val, ekey, newval))
+        alterdata.renameVar(key, ekey)
+
+def inheritFromOS(d, savedenv, permitted):
+    """Inherit variables from the initial environment."""
+    exportlist = bb.utils.preserved_envvars_exported()
+    for s in savedenv.keys():
+        if s in permitted:
+            try:
+                d.setVar(s, savedenv.getVar(s, True), op = 'from env')
+                if s in exportlist:
+                    d.setVarFlag(s, "export", True, op = 'auto env export')
+            except TypeError:
+                pass
+
+def emit_var(var, o=sys.__stdout__, d = init(), all=False):
+    """Emit a variable to be sourced by a shell."""
+    if d.getVarFlag(var, "python", False):
+        return False
+
+    export = d.getVarFlag(var, "export", False)
+    unexport = d.getVarFlag(var, "unexport", False)
+    func = d.getVarFlag(var, "func", False)
+    if not all and not export and not unexport and not func:
+        return False
+
+    try:
+        if all:
+            oval = d.getVar(var, False)
+        val = d.getVar(var, True)
+    except (KeyboardInterrupt, bb.build.FuncFailed):
+        raise
+    except Exception as exc:
+        o.write('# expansion of %s threw %s: %s\n' % (var, exc.__class__.__name__, str(exc)))
+        return False
+
+    if all:
+        d.varhistory.emit(var, oval, val, o, d)
+
+    if (var.find("-") != -1 or var.find(".") != -1 or var.find('{') != -1 or var.find('}') != -1 or var.find('+') != -1) and not all:
+        return False
+
+    varExpanded = d.expand(var)
+
+    if unexport:
+        o.write('unset %s\n' % varExpanded)
+        return False
+
+    if val is None:
+        return False
+
+    val = str(val)
+
+    if varExpanded.startswith("BASH_FUNC_"):
+        varExpanded = varExpanded[10:-2]
+        val = val[3:] # Strip off "() "
+        o.write("%s() %s\n" % (varExpanded, val))
+        o.write("export -f %s\n" % (varExpanded))
+        return True
+
+    if func:
+        # NOTE: should probably check for unbalanced {} within the var
+        val = val.rstrip('\n')
+        o.write("%s() {\n%s\n}\n" % (varExpanded, val))
+        return 1
+
+    if export:
+        o.write('export ')
+
+    # if we're going to output this within doublequotes,
+    # to a shell, we need to escape the quotes in the var
+    alter = re.sub('"', '\\"', val)
+    alter = re.sub('\n', ' \\\n', alter)
+    alter = re.sub('\\$', '\\\\$', alter)
+    o.write('%s="%s"\n' % (varExpanded, alter))
+    return False
+
+def emit_env(o=sys.__stdout__, d = init(), all=False):
+    """Emits all items in the data store in a format such that it can be sourced by a shell."""
+
+    isfunc = lambda key: bool(d.getVarFlag(key, "func", False))
+    keys = sorted((key for key in d.keys() if not key.startswith("__")), key=isfunc)
+    grouped = groupby(keys, isfunc)
+    for isfunc, keys in grouped:
+        for key in keys:
+            emit_var(key, o, d, all and not isfunc) and o.write('\n')
+
+def exported_keys(d):
+    return (key for key in d.keys() if not key.startswith('__') and
+                                      d.getVarFlag(key, 'export', False) and
+                                      not d.getVarFlag(key, 'unexport', False))
+
+def exported_vars(d):
+    for key in exported_keys(d):
+        try:
+            value = d.getVar(key, True)
+        except Exception:
+            pass
+
+        if value is not None:
+            yield key, str(value)
+
+def emit_func(func, o=sys.__stdout__, d = init()):
+    """Emits all items in the data store in a format such that it can be sourced by a shell."""
+
+    keys = (key for key in d.keys() if not key.startswith("__") and not d.getVarFlag(key, "func", False))
+    for key in keys:
+        emit_var(key, o, d, False)
+
+    o.write('\n')
+    emit_var(func, o, d, False) and o.write('\n')
+    newdeps = bb.codeparser.ShellParser(func, logger).parse_shell(d.getVar(func, True))
+    newdeps |= set((d.getVarFlag(func, "vardeps", True) or "").split())
+    seen = set()
+    while newdeps:
+        deps = newdeps
+        seen |= deps
+        newdeps = set()
+        for dep in deps:
+            if d.getVarFlag(dep, "func", False) and not d.getVarFlag(dep, "python", False):
+               emit_var(dep, o, d, False) and o.write('\n')
+               newdeps |=  bb.codeparser.ShellParser(dep, logger).parse_shell(d.getVar(dep, True))
+               newdeps |= set((d.getVarFlag(dep, "vardeps", True) or "").split())
+        newdeps -= seen
+
+_functionfmt = """
+def {function}(d):
+{body}"""
+
+def emit_func_python(func, o=sys.__stdout__, d = init()):
+    """Emits all items in the data store in a format such that it can be sourced by a shell."""
+
+    def write_func(func, o, call = False):
+        body = d.getVar(func, False)
+        if not body.startswith("def"):
+            body = _functionfmt.format(function=func, body=body)
+
+        o.write(body.strip() + "\n\n")
+        if call:
+            o.write(func + "(d)" + "\n\n")
+
+    write_func(func, o, True)
+    pp = bb.codeparser.PythonParser(func, logger)
+    pp.parse_python(d.getVar(func, False))
+    newdeps = pp.execs
+    newdeps |= set((d.getVarFlag(func, "vardeps", True) or "").split())
+    seen = set()
+    while newdeps:
+        deps = newdeps
+        seen |= deps
+        newdeps = set()
+        for dep in deps:
+            if d.getVarFlag(dep, "func", False) and d.getVarFlag(dep, "python", False):
+               write_func(dep, o)
+               pp = bb.codeparser.PythonParser(dep, logger)
+               pp.parse_python(d.getVar(dep, False))
+               newdeps |= pp.execs
+               newdeps |= set((d.getVarFlag(dep, "vardeps", True) or "").split())
+        newdeps -= seen
+
+def update_data(d):
+    """Performs final steps upon the datastore, including application of overrides"""
+    d.finalize(parent = True)
+
+def build_dependencies(key, keys, shelldeps, varflagsexcl, d):
+    deps = set()
+    try:
+        if key[-1] == ']':
+            vf = key[:-1].split('[')
+            value = d.getVarFlag(vf[0], vf[1], False)
+            parser = d.expandWithRefs(value, key)
+            deps |= parser.references
+            deps = deps | (keys & parser.execs)
+            return deps, value
+        varflags = d.getVarFlags(key, ["vardeps", "vardepvalue", "vardepsexclude", "vardepvalueexclude", "postfuncs", "prefuncs", "lineno", "filename"]) or {}
+        vardeps = varflags.get("vardeps")
+        value = d.getVar(key, False)
+
+        def handle_contains(value, contains, d):
+            newvalue = ""
+            for k in sorted(contains):
+                l = (d.getVar(k, True) or "").split()
+                for word in sorted(contains[k]):
+                    if word in l:
+                        newvalue += "\n%s{%s} = Set" %  (k, word)
+                    else:
+                        newvalue += "\n%s{%s} = Unset" %  (k, word)
+            if not newvalue:
+                return value
+            if not value:
+                return newvalue
+            return value + newvalue
+
+        if "vardepvalue" in varflags:
+           value = varflags.get("vardepvalue")
+        elif varflags.get("func"):
+            if varflags.get("python"):
+                parser = bb.codeparser.PythonParser(key, logger)
+                if value and "\t" in value:
+                    logger.warn("Variable %s contains tabs, please remove these (%s)" % (key, d.getVar("FILE", True)))
+                parser.parse_python(value, filename=varflags.get("filename"), lineno=varflags.get("lineno"))
+                deps = deps | parser.references
+                deps = deps | (keys & parser.execs)
+                value = handle_contains(value, parser.contains, d)
+            else:
+                parsedvar = d.expandWithRefs(value, key)
+                parser = bb.codeparser.ShellParser(key, logger)
+                parser.parse_shell(parsedvar.value)
+                deps = deps | shelldeps
+                deps = deps | parsedvar.references
+                deps = deps | (keys & parser.execs) | (keys & parsedvar.execs)
+                value = handle_contains(value, parsedvar.contains, d)
+            if vardeps is None:
+                parser.log.flush()
+            if "prefuncs" in varflags:
+                deps = deps | set(varflags["prefuncs"].split())
+            if "postfuncs" in varflags:
+                deps = deps | set(varflags["postfuncs"].split())
+        else:
+            parser = d.expandWithRefs(value, key)
+            deps |= parser.references
+            deps = deps | (keys & parser.execs)
+            value = handle_contains(value, parser.contains, d)
+
+        if "vardepvalueexclude" in varflags:
+            exclude = varflags.get("vardepvalueexclude")
+            for excl in exclude.split('|'):
+                if excl:
+                    value = value.replace(excl, '')
+
+        # Add varflags, assuming an exclusion list is set
+        if varflagsexcl:
+            varfdeps = []
+            for f in varflags:
+                if f not in varflagsexcl:
+                    varfdeps.append('%s[%s]' % (key, f))
+            if varfdeps:
+                deps |= set(varfdeps)
+
+        deps |= set((vardeps or "").split())
+        deps -= set(varflags.get("vardepsexclude", "").split())
+    except Exception as e:
+        bb.warn("Exception during build_dependencies for %s" % key)
+        raise
+    return deps, value
+    #bb.note("Variable %s references %s and calls %s" % (key, str(deps), str(execs)))
+    #d.setVarFlag(key, "vardeps", deps)
+
+def generate_dependencies(d):
+
+    keys = set(key for key in d if not key.startswith("__"))
+    shelldeps = set(key for key in d.getVar("__exportlist", False) if d.getVarFlag(key, "export", False) and not d.getVarFlag(key, "unexport", False))
+    varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS', True)
+
+    deps = {}
+    values = {}
+
+    tasklist = d.getVar('__BBTASKS', False) or []
+    for task in tasklist:
+        deps[task], values[task] = build_dependencies(task, keys, shelldeps, varflagsexcl, d)
+        newdeps = deps[task]
+        seen = set()
+        while newdeps:
+            nextdeps = newdeps
+            seen |= nextdeps
+            newdeps = set()
+            for dep in nextdeps:
+                if dep not in deps:
+                    deps[dep], values[dep] = build_dependencies(dep, keys, shelldeps, varflagsexcl, d)
+                newdeps |=  deps[dep]
+            newdeps -= seen
+        #print "For %s: %s" % (task, str(deps[task]))
+    return tasklist, deps, values
+
+def inherits_class(klass, d):
+    val = d.getVar('__inherit_cache', False) or []
+    needle = os.path.join('classes', '%s.bbclass' % klass)
+    for v in val:
+        if v.endswith(needle):
+            return True
+    return False

bitbake/lib/bb/data_smart.py

@@ -0,0 +1,969 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake Smart Dictionary Implementation
+
+Functions for interacting with the data structure used by the
+BitBake build tools.
+
+"""
+
+# Copyright (C) 2003, 2004  Chris Larson
+# Copyright (C) 2004, 2005  Seb Frankengul
+# Copyright (C) 2005, 2006  Holger Hans Peter Freyther
+# Copyright (C) 2005        Uli Luckas
+# Copyright (C) 2005        ROAD GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+import copy, re, sys, traceback
+from collections import MutableMapping
+import logging
+import hashlib
+import bb, bb.codeparser
+from bb   import utils
+from bb.COW  import COWDictBase
+
+logger = logging.getLogger("BitBake.Data")
+
+__setvar_keyword__ = ["_append", "_prepend", "_remove"]
+__setvar_regexp__ = re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>.*))?$')
+__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
+__expand_python_regexp__ = re.compile(r"\${@.+?}")
+
+def infer_caller_details(loginfo, parent = False, varval = True):
+    """Save the caller the trouble of specifying everything."""
+    # Save effort.
+    if 'ignore' in loginfo and loginfo['ignore']:
+        return
+    # If nothing was provided, mark this as possibly unneeded.
+    if not loginfo:
+        loginfo['ignore'] = True
+        return
+    # Infer caller's likely values for variable (var) and value (value), 
+    # to reduce clutter in the rest of the code.
+    above = None
+    def set_above():
+        try:
+            raise Exception
+        except Exception:
+            tb = sys.exc_info()[2]
+            if parent:
+                return tb.tb_frame.f_back.f_back.f_back
+            else:
+                return tb.tb_frame.f_back.f_back
+
+    if varval and ('variable' not in loginfo or 'detail' not in loginfo):
+        if not above:
+            above = set_above()
+        lcls = above.f_locals.items()
+        for k, v in lcls:
+            if k == 'value' and 'detail' not in loginfo:
+                loginfo['detail'] = v
+            if k == 'var' and 'variable' not in loginfo:
+                loginfo['variable'] = v
+    # Infer file/line/function from traceback
+    # Don't use traceback.extract_stack() since it fills the line contents which
+    # we don't need and that hits stat syscalls
+    if 'file' not in loginfo:
+        if not above:
+            above = set_above()
+        f = above.f_back
+        line = f.f_lineno
+        file = f.f_code.co_filename
+        func = f.f_code.co_name
+        loginfo['file'] = file
+        loginfo['line'] = line
+        if func not in loginfo:
+            loginfo['func'] = func
+
+class VariableParse:
+    def __init__(self, varname, d, val = None):
+        self.varname = varname
+        self.d = d
+        self.value = val
+
+        self.references = set()
+        self.execs = set()
+        self.contains = {}
+
+    def var_sub(self, match):
+            key = match.group()[2:-1]
+            if self.varname and key:
+                if self.varname == key:
+                    raise Exception("variable %s references itself!" % self.varname)
+            if key in self.d.expand_cache:
+                varparse = self.d.expand_cache[key]
+                var = varparse.value
+            else:
+                var = self.d.getVarFlag(key, "_content", True)
+            self.references.add(key)
+            if var is not None:
+                return var
+            else:
+                return match.group()
+
+    def python_sub(self, match):
+            code = match.group()[3:-1]
+            codeobj = compile(code.strip(), self.varname or "<expansion>", "eval")
+
+            parser = bb.codeparser.PythonParser(self.varname, logger)
+            parser.parse_python(code)
+            if self.varname:
+                vardeps = self.d.getVarFlag(self.varname, "vardeps", True)
+                if vardeps is None:
+                    parser.log.flush()
+            else:
+                parser.log.flush()
+            self.references |= parser.references
+            self.execs |= parser.execs
+
+            for k in parser.contains:
+                if k not in self.contains:
+                    self.contains[k] = parser.contains[k].copy()
+                else:
+                    self.contains[k].update(parser.contains[k])
+            value = utils.better_eval(codeobj, DataContext(self.d))
+            return str(value)
+
+
+class DataContext(dict):
+    def __init__(self, metadata, **kwargs):
+        self.metadata = metadata
+        dict.__init__(self, **kwargs)
+        self['d'] = metadata
+
+    def __missing__(self, key):
+        value = self.metadata.getVar(key, True)
+        if value is None or self.metadata.getVarFlag(key, 'func', False):
+            raise KeyError(key)
+        else:
+            return value
+
+class ExpansionError(Exception):
+    def __init__(self, varname, expression, exception):
+        self.expression = expression
+        self.variablename = varname
+        self.exception = exception
+        if varname:
+            if expression:
+                self.msg = "Failure expanding variable %s, expression was %s which triggered exception %s: %s" % (varname, expression, type(exception).__name__, exception)
+            else:
+                self.msg = "Failure expanding variable %s: %s: %s" % (varname, type(exception).__name__, exception)
+        else:
+            self.msg = "Failure expanding expression %s which triggered exception %s: %s" % (expression, type(exception).__name__, exception)
+        Exception.__init__(self, self.msg)
+        self.args = (varname, expression, exception)
+    def __str__(self):
+        return self.msg
+
+class IncludeHistory(object):
+    def __init__(self, parent = None, filename = '[TOP LEVEL]'):
+        self.parent = parent
+        self.filename = filename
+        self.children = []
+        self.current = self
+
+    def copy(self):
+        new = IncludeHistory(self.parent, self.filename)
+        for c in self.children:
+            new.children.append(c)
+        return new
+
+    def include(self, filename):
+        newfile = IncludeHistory(self.current, filename)
+        self.current.children.append(newfile)
+        self.current = newfile
+        return self
+
+    def __enter__(self):
+        pass
+
+    def __exit__(self, a, b, c):
+        if self.current.parent:
+            self.current = self.current.parent
+        else:
+            bb.warn("Include log: Tried to finish '%s' at top level." % filename)
+        return False
+
+    def emit(self, o, level = 0):
+        """Emit an include history file, and its children."""
+        if level:
+            spaces = "  " * (level - 1)
+            o.write("# %s%s" % (spaces, self.filename))
+            if len(self.children) > 0:
+                o.write(" includes:")
+        else:
+            o.write("#\n# INCLUDE HISTORY:\n#")
+        level = level + 1
+        for child in self.children:
+            o.write("\n")
+            child.emit(o, level)
+
+class VariableHistory(object):
+    def __init__(self, dataroot):
+        self.dataroot = dataroot
+        self.variables = COWDictBase.copy()
+
+    def copy(self):
+        new = VariableHistory(self.dataroot)
+        new.variables = self.variables.copy()
+        return new
+
+    def record(self, *kwonly, **loginfo):
+        if not self.dataroot._tracking:
+            return
+        if len(kwonly) > 0:
+            raise TypeError
+        infer_caller_details(loginfo, parent = True)
+        if 'ignore' in loginfo and loginfo['ignore']:
+            return
+        if 'op' not in loginfo or not loginfo['op']:
+            loginfo['op'] = 'set'
+        if 'detail' in loginfo:
+            loginfo['detail'] = str(loginfo['detail'])
+        if 'variable' not in loginfo or 'file' not in loginfo:
+            raise ValueError("record() missing variable or file.")
+        var = loginfo['variable']
+
+        if var not in self.variables:
+            self.variables[var] = []
+        if not isinstance(self.variables[var], list):
+            return
+        if 'nodups' in loginfo and loginfo in self.variables[var]:
+            return
+        self.variables[var].append(loginfo.copy())
+
+    def variable(self, var):
+        if var in self.variables:
+            return self.variables[var]
+        else:
+            return []
+
+    def emit(self, var, oval, val, o, d):
+        history = self.variable(var)
+
+        # Append override history
+        if var in d.overridedata:
+            for (r, override) in d.overridedata[var]:
+                for event in self.variable(r):
+                    loginfo = event.copy()
+                    if 'flag' in loginfo and not loginfo['flag'].startswith("_"):
+                        continue
+                    loginfo['variable'] = var
+                    loginfo['op'] = 'override[%s]:%s' % (override, loginfo['op'])
+                    history.append(loginfo)
+
+        commentVal = re.sub('\n', '\n#', str(oval))
+        if history:
+            if len(history) == 1:
+                o.write("#\n# $%s\n" % var)
+            else:
+                o.write("#\n# $%s [%d operations]\n" % (var, len(history)))
+            for event in history:
+                # o.write("# %s\n" % str(event))
+                if 'func' in event:
+                    # If we have a function listed, this is internal
+                    # code, not an operation in a config file, and the
+                    # full path is distracting.
+                    event['file'] = re.sub('.*/', '', event['file'])
+                    display_func = ' [%s]' % event['func']
+                else:
+                    display_func = ''
+                if 'flag' in event:
+                    flag = '[%s] ' % (event['flag'])
+                else:
+                    flag = ''
+                o.write("#   %s %s:%s%s\n#     %s\"%s\"\n" % (event['op'], event['file'], event['line'], display_func, flag, re.sub('\n', '\n#     ', event['detail'])))
+            if len(history) > 1:
+                o.write("# pre-expansion value:\n")
+                o.write('#   "%s"\n' % (commentVal))
+        else:
+            o.write("#\n# $%s\n#   [no history recorded]\n#\n" % var)
+            o.write('#   "%s"\n' % (commentVal))
+
+    def get_variable_files(self, var):
+        """Get the files where operations are made on a variable"""
+        var_history = self.variable(var)
+        files = []
+        for event in var_history:
+            files.append(event['file'])
+        return files
+
+    def get_variable_lines(self, var, f):
+        """Get the line where a operation is made on a variable in file f"""
+        var_history = self.variable(var)
+        lines = []
+        for event in var_history:
+            if f== event['file']:
+                line = event['line']
+                lines.append(line)
+        return lines
+
+    def get_variable_items_files(self, var, d):
+        """
+        Use variable history to map items added to a list variable and
+        the files in which they were added.
+        """
+        history = self.variable(var)
+        finalitems = (d.getVar(var, True) or '').split()
+        filemap = {}
+        isset = False
+        for event in history:
+            if 'flag' in event:
+                continue
+            if event['op'] == '_remove':
+                continue
+            if isset and event['op'] == 'set?':
+                continue
+            isset = True
+            items = d.expand(event['detail']).split()
+            for item in items:
+                # This is a little crude but is belt-and-braces to avoid us
+                # having to handle every possible operation type specifically
+                if item in finalitems and not item in filemap:
+                    filemap[item] = event['file']
+        return filemap
+
+    def del_var_history(self, var, f=None, line=None):
+        """If file f and line are not given, the entire history of var is deleted"""
+        if var in self.variables:
+            if f and line:
+                self.variables[var] = [ x for x in self.variables[var] if x['file']!=f and x['line']!=line]
+            else:
+                self.variables[var] = []
+
+class DataSmart(MutableMapping):
+    def __init__(self):
+        self.dict = {}
+
+        self.inchistory = IncludeHistory()
+        self.varhistory = VariableHistory(self)
+        self._tracking = False
+
+        self.expand_cache = {}
+
+        # cookie monster tribute
+        # Need to be careful about writes to overridedata as
+        # its only a shallow copy, could influence other data store
+        # copies!
+        self.overridedata = {}
+        self.overrides = None
+        self.overridevars = set(["OVERRIDES", "FILE"])
+        self.inoverride = False
+
+    def enableTracking(self):
+        self._tracking = True
+
+    def disableTracking(self):
+        self._tracking = False
+
+    def expandWithRefs(self, s, varname):
+
+        if not isinstance(s, basestring): # sanity check
+            return VariableParse(varname, self, s)
+
+        if varname and varname in self.expand_cache:
+            return self.expand_cache[varname]
+
+        varparse = VariableParse(varname, self)
+
+        while s.find('${') != -1:
+            olds = s
+            try:
+                s = __expand_var_regexp__.sub(varparse.var_sub, s)
+                try:
+                    s = __expand_python_regexp__.sub(varparse.python_sub, s)
+                except SyntaxError as e:
+                    # Likely unmatched brackets, just don't expand the expression
+                    if e.msg != "EOL while scanning string literal":
+                        raise
+                if s == olds:
+                    break
+            except ExpansionError:
+                raise
+            except bb.parse.SkipRecipe:
+                raise
+            except Exception as exc:
+                exc_class, exc, tb = sys.exc_info()
+                raise ExpansionError, ExpansionError(varname, s, exc), tb
+
+        varparse.value = s
+
+        if varname:
+            self.expand_cache[varname] = varparse
+
+        return varparse
+
+    def expand(self, s, varname = None):
+        return self.expandWithRefs(s, varname).value
+
+    def finalize(self, parent = False):
+        return
+
+    def internal_finalize(self, parent = False):
+        """Performs final steps upon the datastore, including application of overrides"""
+        self.overrides = None
+
+    def need_overrides(self):
+        if self.overrides is not None:
+            return
+        if self.inoverride:
+            return
+        for count in range(5):
+            self.inoverride = True
+            # Can end up here recursively so setup dummy values
+            self.overrides = []
+            self.overridesset = set()
+            self.overrides = (self.getVar("OVERRIDES", True) or "").split(":") or []
+            self.overridesset = set(self.overrides)
+            self.inoverride = False
+            self.expand_cache = {}
+            newoverrides = (self.getVar("OVERRIDES", True) or "").split(":") or []
+            if newoverrides == self.overrides:
+                break
+            self.overrides = newoverrides
+            self.overridesset = set(self.overrides)
+        else:
+            bb.fatal("Overrides could not be expanded into a stable state after 5 iterations, overrides must be being referenced by other overridden variables in some recursive fashion. Please provide your configuration to bitbake-devel so we can laugh, er, I mean try and understand how to make it work.")
+
+    def initVar(self, var):
+        self.expand_cache = {}
+        if not var in self.dict:
+            self.dict[var] = {}
+
+    def _findVar(self, var):
+        dest = self.dict
+        while dest:
+            if var in dest:
+                return dest[var]
+
+            if "_data" not in dest:
+                break
+            dest = dest["_data"]
+
+    def _makeShadowCopy(self, var):
+        if var in self.dict:
+            return
+
+        local_var = self._findVar(var)
+
+        if local_var:
+            self.dict[var] = copy.copy(local_var)
+        else:
+            self.initVar(var)
+
+
+    def setVar(self, var, value, **loginfo):
+        #print("var=" + str(var) + "  val=" + str(value))
+        parsing=False
+        if 'parsing' in loginfo:
+            parsing=True
+
+        if 'op' not in loginfo:
+            loginfo['op'] = "set"
+        self.expand_cache = {}
+        match  = __setvar_regexp__.match(var)
+        if match and match.group("keyword") in __setvar_keyword__:
+            base = match.group('base')
+            keyword = match.group("keyword")
+            override = match.group('add')
+            l = self.getVarFlag(base, keyword, False) or []
+            l.append([value, override])
+            self.setVarFlag(base, keyword, l, ignore=True)
+            # And cause that to be recorded:
+            loginfo['detail'] = value
+            loginfo['variable'] = base
+            if override:
+                loginfo['op'] = '%s[%s]' % (keyword, override)
+            else:
+                loginfo['op'] = keyword
+            self.varhistory.record(**loginfo)
+            # todo make sure keyword is not __doc__ or __module__
+            # pay the cookie monster
+
+            # more cookies for the cookie monster
+            if '_' in var:
+                self._setvar_update_overrides(base, **loginfo)
+
+            if base in self.overridevars:
+                self._setvar_update_overridevars(var, value)
+            return
+
+        if not var in self.dict:
+            self._makeShadowCopy(var)
+
+        if not parsing:
+            if "_append" in self.dict[var]:
+                del self.dict[var]["_append"]
+            if "_prepend" in self.dict[var]:
+                del self.dict[var]["_prepend"]
+            if var in self.overridedata:
+                active = []
+                self.need_overrides()
+                for (r, o) in self.overridedata[var]:
+                    if o in self.overridesset:
+                        active.append(r)
+                    elif "_" in o:
+                        if set(o.split("_")).issubset(self.overridesset):
+                            active.append(r)
+                for a in active:
+                    self.delVar(a)
+                del self.overridedata[var]
+
+        # more cookies for the cookie monster
+        if '_' in var:
+            self._setvar_update_overrides(var, **loginfo)
+
+        # setting var
+        self.dict[var]["_content"] = value
+        self.varhistory.record(**loginfo)
+
+        if var in self.overridevars:
+            self._setvar_update_overridevars(var, value)
+
+    def _setvar_update_overridevars(self, var, value):
+        vardata = self.expandWithRefs(value, var)
+        new = vardata.references
+        new.update(vardata.contains.keys())
+        while not new.issubset(self.overridevars):
+            nextnew = set()
+            self.overridevars.update(new)
+            for i in new:
+                vardata = self.expandWithRefs(self.getVar(i, True), i)
+                nextnew.update(vardata.references)
+                nextnew.update(vardata.contains.keys())
+            new = nextnew
+        self.internal_finalize(True)
+
+    def _setvar_update_overrides(self, var, **loginfo):
+        # aka pay the cookie monster
+        override = var[var.rfind('_')+1:]
+        shortvar = var[:var.rfind('_')]
+        while override and override.islower():
+            if shortvar not in self.overridedata:
+                self.overridedata[shortvar] = []
+            if [var, override] not in self.overridedata[shortvar]:
+                # Force CoW by recreating the list first
+                self.overridedata[shortvar] = list(self.overridedata[shortvar])
+                self.overridedata[shortvar].append([var, override])
+            override = None
+            if "_" in shortvar:
+                override = var[shortvar.rfind('_')+1:]
+                shortvar = var[:shortvar.rfind('_')]
+                if len(shortvar) == 0:
+                    override = None
+
+    def getVar(self, var, expand, noweakdefault=False, parsing=False):
+        return self.getVarFlag(var, "_content", expand, noweakdefault, parsing)
+
+    def renameVar(self, key, newkey, **loginfo):
+        """
+        Rename the variable key to newkey
+        """
+        val = self.getVar(key, 0, parsing=True)
+        if val is not None:
+            loginfo['variable'] = newkey
+            loginfo['op'] = 'rename from %s' % key
+            loginfo['detail'] = val
+            self.varhistory.record(**loginfo)
+            self.setVar(newkey, val, ignore=True, parsing=True)
+
+        for i in (__setvar_keyword__):
+            src = self.getVarFlag(key, i, False)
+            if src is None:
+                continue
+
+            dest = self.getVarFlag(newkey, i, False) or []
+            dest.extend(src)
+            self.setVarFlag(newkey, i, dest, ignore=True)
+
+        if key in self.overridedata:
+            self.overridedata[newkey] = []
+            for (v, o) in self.overridedata[key]:
+                self.overridedata[newkey].append([v.replace(key, newkey), o])
+                self.renameVar(v, v.replace(key, newkey))
+
+        if '_' in newkey and val is None:
+            self._setvar_update_overrides(newkey, **loginfo)
+
+        loginfo['variable'] = key
+        loginfo['op'] = 'rename (to)'
+        loginfo['detail'] = newkey
+        self.varhistory.record(**loginfo)
+        self.delVar(key, ignore=True)
+
+    def appendVar(self, var, value, **loginfo):
+        loginfo['op'] = 'append'
+        self.varhistory.record(**loginfo)
+        self.setVar(var + "_append", value, ignore=True, parsing=True)
+
+    def prependVar(self, var, value, **loginfo):
+        loginfo['op'] = 'prepend'
+        self.varhistory.record(**loginfo)
+        self.setVar(var + "_prepend", value, ignore=True, parsing=True)
+
+    def delVar(self, var, **loginfo):
+        loginfo['detail'] = ""
+        loginfo['op'] = 'del'
+        self.varhistory.record(**loginfo)
+        self.expand_cache = {}
+        self.dict[var] = {}
+        if var in self.overridedata:
+            del self.overridedata[var]
+        if '_' in var:
+            override = var[var.rfind('_')+1:]
+            shortvar = var[:var.rfind('_')]
+            while override and override.islower():
+                try:
+                    if shortvar in self.overridedata:
+                        # Force CoW by recreating the list first
+                        self.overridedata[shortvar] = list(self.overridedata[shortvar])
+                        self.overridedata[shortvar].remove([var, override])
+                except ValueError as e:
+                    pass
+                override = None
+                if "_" in shortvar:
+                    override = var[shortvar.rfind('_')+1:]
+                    shortvar = var[:shortvar.rfind('_')]
+                    if len(shortvar) == 0:
+                         override = None
+
+    def setVarFlag(self, var, flag, value, **loginfo):
+        self.expand_cache = {}
+        if 'op' not in loginfo:
+            loginfo['op'] = "set"
+        loginfo['flag'] = flag
+        self.varhistory.record(**loginfo)
+        if not var in self.dict:
+            self._makeShadowCopy(var)
+        self.dict[var][flag] = value
+
+        if flag == "_defaultval" and '_' in var:
+            self._setvar_update_overrides(var, **loginfo)
+        if flag == "_defaultval" and var in self.overridevars:
+            self._setvar_update_overridevars(var, value)
+
+        if flag == "unexport" or flag == "export":
+            if not "__exportlist" in self.dict:
+                self._makeShadowCopy("__exportlist")
+            if not "_content" in self.dict["__exportlist"]:
+                self.dict["__exportlist"]["_content"] = set()
+            self.dict["__exportlist"]["_content"].add(var)
+
+    def getVarFlag(self, var, flag, expand, noweakdefault=False, parsing=False):
+        local_var = self._findVar(var)
+        value = None
+        if flag == "_content" and var in self.overridedata and not parsing:
+            match = False
+            active = {}
+            self.need_overrides()
+            for (r, o) in self.overridedata[var]:
+                # What about double overrides both with "_" in the name?
+                if o in self.overridesset:
+                    active[o] = r
+                elif "_" in o:
+                    if set(o.split("_")).issubset(self.overridesset):
+                        active[o] = r
+
+            mod = True
+            while mod:
+                mod = False
+                for o in self.overrides:
+                    for a in active.copy():
+                        if a.endswith("_" + o):
+                            t = active[a]
+                            del active[a]
+                            active[a.replace("_" + o, "")] = t
+                            mod = True
+                        elif a == o:
+                            match = active[a]
+                            del active[a]
+            if match:
+                value = self.getVar(match, False)
+
+        if local_var is not None and value is None:
+            if flag in local_var:
+                value = copy.copy(local_var[flag])
+            elif flag == "_content" and "_defaultval" in local_var and not noweakdefault:
+                value = copy.copy(local_var["_defaultval"])
+
+
+        if flag == "_content" and local_var is not None and "_append" in local_var and not parsing:
+            if not value:
+                value = ""
+            self.need_overrides()
+            for (r, o) in local_var["_append"]:
+                match = True
+                if o:
+                    for o2 in o.split("_"):
+                        if not o2 in self.overrides:
+                            match = False                            
+                if match:
+                    value = value + r
+
+        if flag == "_content" and local_var is not None and "_prepend" in local_var and not parsing:
+            if not value:
+                value = ""
+            self.need_overrides()
+            for (r, o) in local_var["_prepend"]:
+
+                match = True
+                if o:
+                    for o2 in o.split("_"):
+                        if not o2 in self.overrides:
+                            match = False                            
+                if match:
+                    value = r + value
+
+        if expand and value:
+            # Only getvar (flag == _content) hits the expand cache
+            cachename = None
+            if flag == "_content":
+                cachename = var
+            else:
+                cachename = var + "[" + flag + "]"
+            value = self.expand(value, cachename)
+
+        if value and flag == "_content" and local_var is not None and "_remove" in local_var:
+            removes = []
+            self.need_overrides()
+            for (r, o) in local_var["_remove"]:
+                match = True
+                if o:
+                    for o2 in o.split("_"):
+                        if not o2 in self.overrides:
+                            match = False                            
+                if match:
+                    removes.extend(self.expand(r).split())
+
+            filtered = filter(lambda v: v not in removes,
+                              value.split())
+            value = " ".join(filtered)
+            if expand and var in self.expand_cache:
+                 # We need to ensure the expand cache has the correct value
+                 # flag == "_content" here
+                self.expand_cache[var].value = value
+        return value
+
+    def delVarFlag(self, var, flag, **loginfo):
+        self.expand_cache = {}
+        local_var = self._findVar(var)
+        if not local_var:
+            return
+        if not var in self.dict:
+            self._makeShadowCopy(var)
+
+        if var in self.dict and flag in self.dict[var]:
+            loginfo['detail'] = ""
+            loginfo['op'] = 'delFlag'
+            loginfo['flag'] = flag
+            self.varhistory.record(**loginfo)
+
+            del self.dict[var][flag]
+
+    def appendVarFlag(self, var, flag, value, **loginfo):
+        loginfo['op'] = 'append'
+        loginfo['flag'] = flag
+        self.varhistory.record(**loginfo)
+        newvalue = (self.getVarFlag(var, flag, False) or "") + value
+        self.setVarFlag(var, flag, newvalue, ignore=True)
+
+    def prependVarFlag(self, var, flag, value, **loginfo):
+        loginfo['op'] = 'prepend'
+        loginfo['flag'] = flag
+        self.varhistory.record(**loginfo)
+        newvalue = value + (self.getVarFlag(var, flag, False) or "")
+        self.setVarFlag(var, flag, newvalue, ignore=True)
+
+    def setVarFlags(self, var, flags, **loginfo):
+        self.expand_cache = {}
+        infer_caller_details(loginfo)
+        if not var in self.dict:
+            self._makeShadowCopy(var)
+
+        for i in flags:
+            if i == "_content":
+                continue
+            loginfo['flag'] = i
+            loginfo['detail'] = flags[i]
+            self.varhistory.record(**loginfo)
+            self.dict[var][i] = flags[i]
+
+    def getVarFlags(self, var, expand = False, internalflags=False):
+        local_var = self._findVar(var)
+        flags = {}
+
+        if local_var:
+            for i in local_var:
+                if i.startswith("_") and not internalflags:
+                    continue
+                flags[i] = local_var[i]
+                if expand and i in expand:
+                    flags[i] = self.expand(flags[i], var + "[" + i + "]")
+        if len(flags) == 0:
+            return None
+        return flags
+
+
+    def delVarFlags(self, var, **loginfo):
+        self.expand_cache = {}
+        if not var in self.dict:
+            self._makeShadowCopy(var)
+
+        if var in self.dict:
+            content = None
+
+            loginfo['op'] = 'delete flags'
+            self.varhistory.record(**loginfo)
+
+            # try to save the content
+            if "_content" in self.dict[var]:
+                content  = self.dict[var]["_content"]
+                self.dict[var]            = {}
+                self.dict[var]["_content"] = content
+            else:
+                del self.dict[var]
+
+    def createCopy(self):
+        """
+        Create a copy of self by setting _data to self
+        """
+        # we really want this to be a DataSmart...
+        data = DataSmart()
+        data.dict["_data"] = self.dict
+        data.varhistory = self.varhistory.copy()
+        data.varhistory.datasmart = data
+        data.inchistory = self.inchistory.copy()
+
+        data._tracking = self._tracking
+
+        data.overrides = None
+        data.overridevars = copy.copy(self.overridevars)
+        # Should really be a deepcopy but has heavy overhead.
+        # Instead, we're careful with writes.
+        data.overridedata = copy.copy(self.overridedata)
+
+        return data
+
+    def expandVarref(self, variable, parents=False):
+        """Find all references to variable in the data and expand it
+           in place, optionally descending to parent datastores."""
+
+        if parents:
+            keys = iter(self)
+        else:
+            keys = self.localkeys()
+
+        ref = '${%s}' % variable
+        value = self.getVar(variable, False)
+        for key in keys:
+            referrervalue = self.getVar(key, False)
+            if referrervalue and ref in referrervalue:
+                self.setVar(key, referrervalue.replace(ref, value))
+
+    def localkeys(self):
+        for key in self.dict:
+            if key != '_data':
+                yield key
+
+    def __iter__(self):
+        deleted = set()
+        overrides = set()
+        def keylist(d):        
+            klist = set()
+            for key in d:
+                if key == "_data":
+                    continue
+                if key in deleted:
+                    continue
+                if key in overrides:
+                    continue
+                if not d[key]:
+                    deleted.add(key)
+                    continue
+                klist.add(key)
+
+            if "_data" in d:
+                klist |= keylist(d["_data"])
+
+            return klist
+
+        self.need_overrides()
+        for var in self.overridedata:
+            for (r, o) in self.overridedata[var]:
+                if o in self.overridesset:
+                    overrides.add(var)
+                elif "_" in o:
+                    if set(o.split("_")).issubset(self.overridesset):
+                        overrides.add(var)
+
+        for k in keylist(self.dict):
+             yield k
+
+        for k in overrides:
+             yield k
+
+    def __len__(self):
+        return len(frozenset(self))
+
+    def __getitem__(self, item):
+        value = self.getVar(item, False)
+        if value is None:
+            raise KeyError(item)
+        else:
+            return value
+
+    def __setitem__(self, var, value):
+        self.setVar(var, value)
+
+    def __delitem__(self, var):
+        self.delVar(var)
+
+    def get_hash(self):
+        data = {}
+        d = self.createCopy()
+        bb.data.expandKeys(d)
+        bb.data.update_data(d)
+
+        config_whitelist = set((d.getVar("BB_HASHCONFIG_WHITELIST", True) or "").split())
+        keys = set(key for key in iter(d) if not key.startswith("__"))
+        for key in keys:
+            if key in config_whitelist:
+                continue
+
+            value = d.getVar(key, False) or ""
+            data.update({key:value})
+
+            varflags = d.getVarFlags(key, internalflags = True)
+            if not varflags:
+                continue
+            for f in varflags:
+                if f == "_content":
+                    continue
+                data.update({'%s[%s]' % (key, f):varflags[f]})
+
+        for key in ["__BBTASKS", "__BBANONFUNCS", "__BBHANDLERS"]:
+            bb_list = d.getVar(key, False) or []
+            bb_list.sort()
+            data.update({key:str(bb_list)})
+
+            if key == "__BBANONFUNCS":
+                for i in bb_list:
+                    value = d.getVar(i, False) or ""
+                    data.update({i:value})
+
+        data_str = str([(k, data[k]) for k in sorted(data.keys())])
+        return hashlib.md5(data_str).hexdigest()

bitbake/lib/bb/event.py

@@ -0,0 +1,679 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Event' implementation
+
+Classes and functions for manipulating 'events' in the
+BitBake build tools.
+"""
+
+# Copyright (C) 2003, 2004  Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os, sys
+import warnings
+try:
+    import cPickle as pickle
+except ImportError:
+    import pickle
+import logging
+import atexit
+import traceback
+import ast
+import bb.utils
+import bb.compat
+import bb.exceptions
+
+# This is the pid for which we should generate the event. This is set when
+# the runqueue forks off.
+worker_pid = 0
+worker_fire = None
+
+logger = logging.getLogger('BitBake.Event')
+
+class Event(object):
+    """Base class for events"""
+
+    def __init__(self):
+        self.pid = worker_pid
+
+Registered        = 10
+AlreadyRegistered = 14
+
+def get_class_handlers():
+    return _handlers
+
+def set_class_handlers(h):
+    global _handlers
+    _handlers = h
+
+def clean_class_handlers():
+    return bb.compat.OrderedDict()
+
+# Internal
+_handlers = clean_class_handlers()
+_ui_handlers = {}
+_ui_logfilters = {}
+_ui_handler_seq = 0
+_event_handler_map = {}
+_catchall_handlers = {}
+_eventfilter = None
+_uiready = False
+
+def execute_handler(name, handler, event, d):
+    event.data = d
+    addedd = False
+    if 'd' not in __builtins__:
+        __builtins__['d'] = d
+        addedd = True
+    try:
+        ret = handler(event)
+    except (bb.parse.SkipRecipe, bb.BBHandledException):
+        raise
+    except Exception:
+        etype, value, tb = sys.exc_info()
+        logger.error("Execution of event handler '%s' failed" % name,
+                        exc_info=(etype, value, tb.tb_next))
+        raise
+    except SystemExit as exc:
+        if exc.code != 0:
+            logger.error("Execution of event handler '%s' failed" % name)
+        raise
+    finally:
+        del event.data
+        if addedd:
+            del __builtins__['d']
+
+def fire_class_handlers(event, d):
+    if isinstance(event, logging.LogRecord):
+        return
+
+    eid = str(event.__class__)[8:-2]
+    evt_hmap = _event_handler_map.get(eid, {})
+    for name, handler in _handlers.iteritems():
+        if name in _catchall_handlers or name in evt_hmap:
+            if _eventfilter:
+                if not _eventfilter(name, handler, event, d):
+                    continue
+            execute_handler(name, handler, event, d)
+
+ui_queue = []
+@atexit.register
+def print_ui_queue():
+    """If we're exiting before a UI has been spawned, display any queued
+    LogRecords to the console."""
+    logger = logging.getLogger("BitBake")
+    if not _uiready:
+        from bb.msg import BBLogFormatter
+        console = logging.StreamHandler(sys.stdout)
+        console.setFormatter(BBLogFormatter("%(levelname)s: %(message)s"))
+        logger.handlers = [console]
+
+        # First check to see if we have any proper messages
+        msgprint = False
+        for event in ui_queue:
+            if isinstance(event, logging.LogRecord):
+                if event.levelno > logging.DEBUG:
+                    logger.handle(event)
+                    msgprint = True
+        if msgprint:
+            return
+
+        # Nope, so just print all of the messages we have (including debug messages)
+        for event in ui_queue:
+            if isinstance(event, logging.LogRecord):
+                logger.handle(event)
+
+def fire_ui_handlers(event, d):
+    if not _uiready:
+        # No UI handlers registered yet, queue up the messages
+        ui_queue.append(event)
+        return
+
+    errors = []
+    for h in _ui_handlers:
+        #print "Sending event %s" % event
+        try:
+             if not _ui_logfilters[h].filter(event):
+                 continue
+             # We use pickle here since it better handles object instances
+             # which xmlrpc's marshaller does not. Events *must* be serializable
+             # by pickle.
+             if hasattr(_ui_handlers[h].event, "sendpickle"):
+                _ui_handlers[h].event.sendpickle((pickle.dumps(event)))
+             else:
+                _ui_handlers[h].event.send(event)
+        except:
+            errors.append(h)
+    for h in errors:
+        del _ui_handlers[h]
+
+def fire(event, d):
+    """Fire off an Event"""
+
+    # We can fire class handlers in the worker process context and this is
+    # desired so they get the task based datastore.
+    # UI handlers need to be fired in the server context so we defer this. They
+    # don't have a datastore so the datastore context isn't a problem.
+
+    fire_class_handlers(event, d)
+    if worker_fire:
+        worker_fire(event, d)
+    else:
+        fire_ui_handlers(event, d)
+
+def fire_from_worker(event, d):
+    fire_ui_handlers(event, d)
+
+noop = lambda _: None
+def register(name, handler, mask=None, filename=None, lineno=None):
+    """Register an Event handler"""
+
+    # already registered
+    if name in _handlers:
+        return AlreadyRegistered
+
+    if handler is not None:
+        # handle string containing python code
+        if isinstance(handler, basestring):
+            tmp = "def %s(e):\n%s" % (name, handler)
+            try:
+                code = bb.methodpool.compile_cache(tmp)
+                if not code:
+                    if filename is None:
+                        filename = "%s(e)" % name
+                    code = compile(tmp, filename, "exec", ast.PyCF_ONLY_AST)
+                    if lineno is not None:
+                        ast.increment_lineno(code, lineno-1)
+                    code = compile(code, filename, "exec")
+                    bb.methodpool.compile_cache_add(tmp, code)
+            except SyntaxError:
+                logger.error("Unable to register event handler '%s':\n%s", name,
+                             ''.join(traceback.format_exc(limit=0)))
+                _handlers[name] = noop
+                return
+            env = {}
+            bb.utils.better_exec(code, env)
+            func = bb.utils.better_eval(name, env)
+            _handlers[name] = func
+        else:
+            _handlers[name] = handler
+
+        if not mask or '*' in mask:
+            _catchall_handlers[name] = True
+        else:
+            for m in mask:
+                if _event_handler_map.get(m, None) is None:
+                    _event_handler_map[m] = {}
+                _event_handler_map[m][name] = True
+
+        return Registered
+
+def remove(name, handler):
+    """Remove an Event handler"""
+    _handlers.pop(name)
+
+def set_eventfilter(func):
+    global _eventfilter
+    _eventfilter = func
+
+def register_UIHhandler(handler, mainui=False):
+    if mainui:
+        global _uiready
+        _uiready = True
+    bb.event._ui_handler_seq = bb.event._ui_handler_seq + 1
+    _ui_handlers[_ui_handler_seq] = handler
+    level, debug_domains = bb.msg.constructLogOptions()
+    _ui_logfilters[_ui_handler_seq] = UIEventFilter(level, debug_domains)
+    return _ui_handler_seq
+
+def unregister_UIHhandler(handlerNum):
+    if handlerNum in _ui_handlers:
+        del _ui_handlers[handlerNum]
+    return
+
+# Class to allow filtering of events and specific filtering of LogRecords *before* we put them over the IPC
+class UIEventFilter(object):
+    def __init__(self, level, debug_domains):
+        self.update(None, level, debug_domains)
+
+    def update(self, eventmask, level, debug_domains):
+        self.eventmask = eventmask
+        self.stdlevel = level
+        self.debug_domains = debug_domains
+
+    def filter(self, event):
+        if isinstance(event, logging.LogRecord):
+            if event.levelno >= self.stdlevel:
+                return True
+            if event.name in self.debug_domains and event.levelno >= self.debug_domains[event.name]:
+                return True
+            return False
+        eid = str(event.__class__)[8:-2]
+        if self.eventmask and eid not in self.eventmask:
+            return False
+        return True
+
+def set_UIHmask(handlerNum, level, debug_domains, mask):
+    if not handlerNum in _ui_handlers:
+        return False
+    if '*' in mask:
+        _ui_logfilters[handlerNum].update(None, level, debug_domains)
+    else:
+        _ui_logfilters[handlerNum].update(mask, level, debug_domains)
+    return True
+
+def getName(e):
+    """Returns the name of a class or class instance"""
+    if getattr(e, "__name__", None) == None:
+        return e.__class__.__name__
+    else:
+        return e.__name__
+
+class OperationStarted(Event):
+    """An operation has begun"""
+    def __init__(self, msg = "Operation Started"):
+        Event.__init__(self)
+        self.msg = msg
+
+class OperationCompleted(Event):
+    """An operation has completed"""
+    def __init__(self, total, msg = "Operation Completed"):
+        Event.__init__(self)
+        self.total = total
+        self.msg = msg
+
+class OperationProgress(Event):
+    """An operation is in progress"""
+    def __init__(self, current, total, msg = "Operation in Progress"):
+        Event.__init__(self)
+        self.current = current
+        self.total = total
+        self.msg = msg + ": %s/%s" % (current, total);
+
+class ConfigParsed(Event):
+    """Configuration Parsing Complete"""
+
+class RecipeEvent(Event):
+    def __init__(self, fn):
+        self.fn = fn
+        Event.__init__(self)
+
+class RecipePreFinalise(RecipeEvent):
+    """ Recipe Parsing Complete but not yet finialised"""
+
+class RecipeParsed(RecipeEvent):
+    """ Recipe Parsing Complete """
+
+class StampUpdate(Event):
+    """Trigger for any adjustment of the stamp files to happen"""
+
+    def __init__(self, targets, stampfns):
+        self._targets = targets
+        self._stampfns = stampfns
+        Event.__init__(self)
+
+    def getStampPrefix(self):
+        return self._stampfns
+
+    def getTargets(self):
+        return self._targets
+
+    stampPrefix = property(getStampPrefix)
+    targets = property(getTargets)
+
+class BuildBase(Event):
+    """Base class for bbmake run events"""
+
+    def __init__(self, n, p, failures = 0):
+        self._name = n
+        self._pkgs = p
+        Event.__init__(self)
+        self._failures = failures
+
+    def getPkgs(self):
+        return self._pkgs
+
+    def setPkgs(self, pkgs):
+        self._pkgs = pkgs
+
+    def getName(self):
+        return self._name
+
+    def setName(self, name):
+        self._name = name
+
+    def getCfg(self):
+        return self.data
+
+    def setCfg(self, cfg):
+        self.data = cfg
+
+    def getFailures(self):
+        """
+        Return the number of failed packages
+        """
+        return self._failures
+
+    pkgs = property(getPkgs, setPkgs, None, "pkgs property")
+    name = property(getName, setName, None, "name property")
+    cfg = property(getCfg, setCfg, None, "cfg property")
+
+
+
+
+
+class BuildStarted(BuildBase, OperationStarted):
+    """bbmake build run started"""
+    def __init__(self, n, p, failures = 0):
+        OperationStarted.__init__(self, "Building Started")
+        BuildBase.__init__(self, n, p, failures)
+
+class BuildCompleted(BuildBase, OperationCompleted):
+    """bbmake build run completed"""
+    def __init__(self, total, n, p, failures=0, interrupted=0):
+        if not failures:
+            OperationCompleted.__init__(self, total, "Building Succeeded")
+        else:
+            OperationCompleted.__init__(self, total, "Building Failed")
+        self._interrupted = interrupted
+        BuildBase.__init__(self, n, p, failures)
+
+class DiskFull(Event):
+    """Disk full case build aborted"""
+    def __init__(self, dev, type, freespace, mountpoint):
+        Event.__init__(self)
+        self._dev = dev
+        self._type = type
+        self._free = freespace
+        self._mountpoint = mountpoint
+
+class NoProvider(Event):
+    """No Provider for an Event"""
+
+    def __init__(self, item, runtime=False, dependees=None, reasons=None, close_matches=None):
+        Event.__init__(self)
+        self._item = item
+        self._runtime = runtime
+        self._dependees = dependees
+        self._reasons = reasons
+        self._close_matches = close_matches
+
+    def getItem(self):
+        return self._item
+
+    def isRuntime(self):
+        return self._runtime
+
+class MultipleProviders(Event):
+    """Multiple Providers"""
+
+    def  __init__(self, item, candidates, runtime = False):
+        Event.__init__(self)
+        self._item = item
+        self._candidates = candidates
+        self._is_runtime = runtime
+
+    def isRuntime(self):
+        """
+        Is this a runtime issue?
+        """
+        return self._is_runtime
+
+    def getItem(self):
+        """
+        The name for the to be build item
+        """
+        return self._item
+
+    def getCandidates(self):
+        """
+        Get the possible Candidates for a PROVIDER.
+        """
+        return self._candidates
+
+class ParseStarted(OperationStarted):
+    """Recipe parsing for the runqueue has begun"""
+    def __init__(self, total):
+        OperationStarted.__init__(self, "Recipe parsing Started")
+        self.total = total
+
+class ParseCompleted(OperationCompleted):
+    """Recipe parsing for the runqueue has completed"""
+    def __init__(self, cached, parsed, skipped, masked, virtuals, errors, total):
+        OperationCompleted.__init__(self, total, "Recipe parsing Completed")
+        self.cached = cached
+        self.parsed = parsed
+        self.skipped = skipped
+        self.virtuals = virtuals
+        self.masked = masked
+        self.errors = errors
+        self.sofar = cached + parsed
+
+class ParseProgress(OperationProgress):
+    """Recipe parsing progress"""
+    def __init__(self, current, total):
+        OperationProgress.__init__(self, current, total, "Recipe parsing")
+
+
+class CacheLoadStarted(OperationStarted):
+    """Loading of the dependency cache has begun"""
+    def __init__(self, total):
+        OperationStarted.__init__(self, "Loading cache Started")
+        self.total = total
+
+class CacheLoadProgress(OperationProgress):
+    """Cache loading progress"""
+    def __init__(self, current, total):
+        OperationProgress.__init__(self, current, total, "Loading cache")
+
+class CacheLoadCompleted(OperationCompleted):
+    """Cache loading is complete"""
+    def __init__(self, total, num_entries):
+        OperationCompleted.__init__(self, total, "Loading cache Completed")
+        self.num_entries = num_entries
+
+class TreeDataPreparationStarted(OperationStarted):
+    """Tree data preparation started"""
+    def __init__(self):
+        OperationStarted.__init__(self, "Preparing tree data Started")
+
+class TreeDataPreparationProgress(OperationProgress):
+    """Tree data preparation is in progress"""
+    def __init__(self, current, total):
+        OperationProgress.__init__(self, current, total, "Preparing tree data")
+
+class TreeDataPreparationCompleted(OperationCompleted):
+    """Tree data preparation completed"""
+    def __init__(self, total):
+        OperationCompleted.__init__(self, total, "Preparing tree data Completed")
+
+class DepTreeGenerated(Event):
+    """
+    Event when a dependency tree has been generated
+    """
+
+    def __init__(self, depgraph):
+        Event.__init__(self)
+        self._depgraph = depgraph
+
+class TargetsTreeGenerated(Event):
+    """
+    Event when a set of buildable targets has been generated
+    """
+    def __init__(self, model):
+        Event.__init__(self)
+        self._model = model
+
+class ReachableStamps(Event):
+    """
+    An event listing all stamps reachable after parsing
+    which the metadata may use to clean up stale data
+    """
+
+    def __init__(self, stamps):
+        Event.__init__(self)
+        self.stamps = stamps
+
+class FilesMatchingFound(Event):
+    """
+    Event when a list of files matching the supplied pattern has
+    been generated
+    """
+    def __init__(self, pattern, matches):
+        Event.__init__(self)
+        self._pattern = pattern
+        self._matches = matches
+
+class CoreBaseFilesFound(Event):
+    """
+    Event when a list of appropriate config files has been generated
+    """
+    def __init__(self, paths):
+        Event.__init__(self)
+        self._paths = paths
+
+class ConfigFilesFound(Event):
+    """
+    Event when a list of appropriate config files has been generated
+    """
+    def __init__(self, variable, values):
+        Event.__init__(self)
+        self._variable = variable
+        self._values = values
+
+class ConfigFilePathFound(Event):
+    """
+    Event when a path for a config file has been found
+    """
+    def __init__(self, path):
+        Event.__init__(self)
+        self._path = path
+
+class MsgBase(Event):
+    """Base class for messages"""
+
+    def __init__(self, msg):
+        self._message = msg
+        Event.__init__(self)
+
+class MsgDebug(MsgBase):
+    """Debug Message"""
+
+class MsgNote(MsgBase):
+    """Note Message"""
+
+class MsgWarn(MsgBase):
+    """Warning Message"""
+
+class MsgError(MsgBase):
+    """Error Message"""
+
+class MsgFatal(MsgBase):
+    """Fatal Message"""
+
+class MsgPlain(MsgBase):
+    """General output"""
+
+class LogExecTTY(Event):
+    """Send event containing program to spawn on tty of the logger"""
+    def __init__(self, msg, prog, sleep_delay, retries):
+        Event.__init__(self)
+        self.msg = msg
+        self.prog = prog
+        self.sleep_delay = sleep_delay
+        self.retries = retries
+
+class LogHandler(logging.Handler):
+    """Dispatch logging messages as bitbake events"""
+
+    def emit(self, record):
+        if record.exc_info:
+            etype, value, tb = record.exc_info
+            if hasattr(tb, 'tb_next'):
+                tb = list(bb.exceptions.extract_traceback(tb, context=3))
+            # Need to turn the value into something the logging system can pickle
+            value = str(value)
+            record.bb_exc_info = (etype, value, tb)
+            record.exc_info = None
+        fire(record, None)
+
+    def filter(self, record):
+        record.taskpid = worker_pid
+        return True
+
+class RequestPackageInfo(Event):
+    """
+    Event to request package information
+    """
+
+class PackageInfo(Event):
+    """
+    Package information for GUI
+    """
+    def __init__(self, pkginfolist):
+        Event.__init__(self)
+        self._pkginfolist = pkginfolist
+
+class MetadataEvent(Event):
+    """
+    Generic event that target for OE-Core classes
+    to report information during asynchrous execution
+    """
+    def __init__(self, eventtype, eventdata):
+        Event.__init__(self)
+        self.type = eventtype
+        self._localdata = eventdata
+
+class SanityCheck(Event):
+    """
+    Event to run sanity checks, either raise errors or generate events as return status.
+    """
+    def __init__(self, generateevents = True):
+        Event.__init__(self)
+        self.generateevents = generateevents
+
+class SanityCheckPassed(Event):
+    """
+    Event to indicate sanity check has passed
+    """
+
+class SanityCheckFailed(Event):
+    """
+    Event to indicate sanity check has failed
+    """
+    def __init__(self, msg, network_error=False):
+        Event.__init__(self)
+        self._msg = msg
+        self._network_error = network_error
+
+class NetworkTest(Event):
+    """
+    Event to run network connectivity tests, either raise errors or generate events as return status.
+    """
+    def __init__(self, generateevents = True):
+        Event.__init__(self)
+        self.generateevents = generateevents
+
+class NetworkTestPassed(Event):
+    """
+    Event to indicate network test has passed
+    """
+
+class NetworkTestFailed(Event):
+    """
+    Event to indicate network test has failed
+    """
+

bitbake/lib/bb/exceptions.py

@@ -0,0 +1,91 @@
+from __future__ import absolute_import
+import inspect
+import traceback
+import bb.namedtuple_with_abc
+from collections import namedtuple
+
+
+class TracebackEntry(namedtuple.abc):
+    """Pickleable representation of a traceback entry"""
+    _fields = 'filename lineno function args code_context index'
+    _header = '  File "{0.filename}", line {0.lineno}, in {0.function}{0.args}'
+
+    def format(self, formatter=None):
+        if not self.code_context:
+            return self._header.format(self) + '\n'
+
+        formatted = [self._header.format(self) + ':\n']
+
+        for lineindex, line in enumerate(self.code_context):
+            if formatter:
+                line = formatter(line)
+
+            if lineindex == self.index:
+                formatted.append('    >%s' % line)
+            else:
+                formatted.append('     %s' % line)
+        return formatted
+
+    def __str__(self):
+        return ''.join(self.format())
+
+def _get_frame_args(frame):
+    """Get the formatted arguments and class (if available) for a frame"""
+    arginfo = inspect.getargvalues(frame)
+
+    try:
+        if not arginfo.args:
+            return '', None
+    # There have been reports from the field of python 2.6 which doesn't 
+    # return a namedtuple here but simply a tuple so fallback gracefully if
+    # args isn't present.
+    except AttributeError:
+        return '', None
+
+    firstarg = arginfo.args[0]
+    if firstarg == 'self':
+        self = arginfo.locals['self']
+        cls = self.__class__.__name__
+
+        arginfo.args.pop(0)
+        del arginfo.locals['self']
+    else:
+        cls = None
+
+    formatted = inspect.formatargvalues(*arginfo)
+    return formatted, cls
+
+def extract_traceback(tb, context=1):
+    frames = inspect.getinnerframes(tb, context)
+    for frame, filename, lineno, function, code_context, index in frames:
+        formatted_args, cls = _get_frame_args(frame)
+        if cls:
+            function = '%s.%s' % (cls, function)
+        yield TracebackEntry(filename, lineno, function, formatted_args,
+                             code_context, index)
+
+def format_extracted(extracted, formatter=None, limit=None):
+    if limit:
+        extracted = extracted[-limit:]
+
+    formatted = []
+    for tracebackinfo in extracted:
+        formatted.extend(tracebackinfo.format(formatter))
+    return formatted
+
+
+def format_exception(etype, value, tb, context=1, limit=None, formatter=None):
+    formatted = ['Traceback (most recent call last):\n']
+
+    if hasattr(tb, 'tb_next'):
+        tb = extract_traceback(tb, context)
+
+    formatted.extend(format_extracted(tb, formatter, limit))
+    formatted.extend(traceback.format_exception_only(etype, value))
+    return formatted
+
+def to_string(exc):
+    if isinstance(exc, SystemExit):
+        if not isinstance(exc.code, basestring):
+            return 'Exited with "%d"' % exc.code
+    return str(exc)

bitbake/lib/bb/fetch2/bzr.py

@@ -0,0 +1,143 @@
+"""
+BitBake 'Fetch' implementation for bzr.
+
+"""
+
+# Copyright (C) 2007 Ross Burton
+# Copyright (C) 2007 Richard Purdie
+#
+#   Classes for obtaining upstream sources for the
+#   BitBake build tools.
+#   Copyright (C) 2003, 2004  Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import sys
+import logging
+import bb
+from bb import data
+from bb.fetch2 import FetchMethod
+from bb.fetch2 import FetchError
+from bb.fetch2 import runfetchcmd
+from bb.fetch2 import logger
+
+class Bzr(FetchMethod):
+    def supports(self, ud, d):
+        return ud.type in ['bzr']
+
+    def urldata_init(self, ud, d):
+        """
+        init bzr specific variable within url data
+        """
+        # Create paths to bzr checkouts
+        relpath = self._strip_leading_slashes(ud.path)
+        ud.pkgdir = os.path.join(data.expand('${BZRDIR}', d), ud.host, relpath)
+
+        ud.setup_revisons(d)
+
+        if not ud.revision:
+            ud.revision = self.latest_revision(ud, d)
+
+        ud.localfile = data.expand('bzr_%s_%s_%s.tar.gz' % (ud.host, ud.path.replace('/', '.'), ud.revision), d)
+
+    def _buildbzrcommand(self, ud, d, command):
+        """
+        Build up an bzr commandline based on ud
+        command is "fetch", "update", "revno"
+        """
+
+        basecmd = data.expand('${FETCHCMD_bzr}', d)
+
+        proto =  ud.parm.get('protocol', 'http')
+
+        bzrroot = ud.host + ud.path
+
+        options = []
+
+        if command == "revno":
+            bzrcmd = "%s revno %s %s://%s" % (basecmd, " ".join(options), proto, bzrroot)
+        else:
+            if ud.revision:
+                options.append("-r %s" % ud.revision)
+
+            if command == "fetch":
+                bzrcmd = "%s branch %s %s://%s" % (basecmd, " ".join(options), proto, bzrroot)
+            elif command == "update":
+                bzrcmd = "%s pull %s --overwrite" % (basecmd, " ".join(options))
+            else:
+                raise FetchError("Invalid bzr command %s" % command, ud.url)
+
+        return bzrcmd
+
+    def download(self, ud, d):
+        """Fetch url"""
+
+        if os.access(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir), '.bzr'), os.R_OK):
+            bzrcmd = self._buildbzrcommand(ud, d, "update")
+            logger.debug(1, "BZR Update %s", ud.url)
+            bb.fetch2.check_network_access(d, bzrcmd, ud.url)
+            os.chdir(os.path.join (ud.pkgdir, os.path.basename(ud.path)))
+            runfetchcmd(bzrcmd, d)
+        else:
+            bb.utils.remove(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir)), True)
+            bzrcmd = self._buildbzrcommand(ud, d, "fetch")
+            bb.fetch2.check_network_access(d, bzrcmd, ud.url)
+            logger.debug(1, "BZR Checkout %s", ud.url)
+            bb.utils.mkdirhier(ud.pkgdir)
+            os.chdir(ud.pkgdir)
+            logger.debug(1, "Running %s", bzrcmd)
+            runfetchcmd(bzrcmd, d)
+
+        os.chdir(ud.pkgdir)
+
+        scmdata = ud.parm.get("scmdata", "")
+        if scmdata == "keep":
+            tar_flags = ""
+        else:
+            tar_flags = "--exclude '.bzr' --exclude '.bzrtags'"
+
+        # tar them up to a defined filename
+        runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.basename(ud.pkgdir)), d, cleanup = [ud.localpath])
+
+    def supports_srcrev(self):
+        return True
+
+    def _revision_key(self, ud, d, name):
+        """
+        Return a unique key for the url
+        """
+        return "bzr:" + ud.pkgdir
+
+    def _latest_revision(self, ud, d, name):
+        """
+        Return the latest upstream revision number
+        """
+        logger.debug(2, "BZR fetcher hitting network for %s", ud.url)
+
+        bb.fetch2.check_network_access(d, self._buildbzrcommand(ud, d, "revno"), ud.url)
+
+        output = runfetchcmd(self._buildbzrcommand(ud, d, "revno"), d, True)
+
+        return output.strip()
+
+    def sortable_revision(self, ud, d, name):
+        """
+        Return a sortable revision number which in our case is the revision number
+        """
+
+        return False, self._build_revision(ud, d)
+
+    def _build_revision(self, ud, d):
+        return ud.revision

bitbake/lib/bb/fetch2/clearcase.py

@@ -0,0 +1,263 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Fetch' clearcase implementation
+
+The clearcase fetcher is used to retrieve files from a ClearCase repository.
+
+Usage in the recipe:
+
+    SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
+    SRCREV = "EXAMPLE_CLEARCASE_TAG"
+    PV = "${@d.getVar("SRCREV", False).replace("/", "+")}"
+
+The fetcher uses the rcleartool or cleartool remote client, depending on which one is available.
+
+Supported SRC_URI options are:
+
+- vob
+    (required) The name of the clearcase VOB (with prepending "/")
+
+- module
+    The module in the selected VOB (with prepending "/")
+
+    The module and vob parameters are combined to create
+    the following load rule in the view config spec:
+                load <vob><module>
+
+- proto
+    http or https
+
+Related variables:
+
+    CCASE_CUSTOM_CONFIG_SPEC
+            Write a config spec to this variable in your recipe to use it instead
+            of the default config spec generated by this fetcher.
+            Please note that the SRCREV loses its functionality if you specify
+            this variable. SRCREV is still used to label the archive after a fetch,
+            but it doesn't define what's fetched.
+
+User credentials:
+    cleartool:
+            The login of cleartool is handled by the system. No special steps needed.
+
+    rcleartool:
+            In order to use rcleartool with authenticated users an `rcleartool login` is
+            necessary before using the fetcher.
+"""
+# Copyright (C) 2014 Siemens AG
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+import os
+import sys
+import shutil
+import bb
+from   bb import data
+from   bb.fetch2 import FetchMethod
+from   bb.fetch2 import FetchError
+from   bb.fetch2 import runfetchcmd
+from   bb.fetch2 import logger
+from   distutils import spawn
+
+class ClearCase(FetchMethod):
+    """Class to fetch urls via 'clearcase'"""
+    def init(self, d):
+        pass
+
+    def supports(self, ud, d):
+        """
+        Check to see if a given url can be fetched with Clearcase.
+        """
+        return ud.type in ['ccrc']
+
+    def debug(self, msg):
+        logger.debug(1, "ClearCase: %s", msg)
+
+    def urldata_init(self, ud, d):
+        """
+        init ClearCase specific variable within url data
+        """
+        ud.proto = "https"
+        if 'protocol' in ud.parm:
+            ud.proto = ud.parm['protocol']
+        if not ud.proto in ('http', 'https'):
+            raise fetch2.ParameterError("Invalid protocol type", ud.url)
+
+        ud.vob = ''
+        if 'vob' in ud.parm:
+            ud.vob = ud.parm['vob']
+        else:
+            msg = ud.url+": vob must be defined so the fetcher knows what to get."
+            raise MissingParameterError('vob', msg)
+
+        if 'module' in ud.parm:
+            ud.module = ud.parm['module']
+        else:
+            ud.module = ""
+
+        ud.basecmd = d.getVar("FETCHCMD_ccrc", True) or spawn.find_executable("cleartool") or spawn.find_executable("rcleartool")
+
+        if data.getVar("SRCREV", d, True) == "INVALID":
+          raise FetchError("Set a valid SRCREV for the clearcase fetcher in your recipe, e.g. SRCREV = \"/main/LATEST\" or any other label of your choice.")
+
+        ud.label = d.getVar("SRCREV", False)
+        ud.customspec = d.getVar("CCASE_CUSTOM_CONFIG_SPEC", True)
+
+        ud.server     = "%s://%s%s" % (ud.proto, ud.host, ud.path)
+
+        ud.identifier = "clearcase-%s%s-%s" % ( ud.vob.replace("/", ""),
+                                                ud.module.replace("/", "."),
+                                                ud.label.replace("/", "."))
+
+        ud.viewname         = "%s-view%s" % (ud.identifier, d.getVar("DATETIME", d, True))
+        ud.csname           = "%s-config-spec" % (ud.identifier)
+        ud.ccasedir         = os.path.join(data.getVar("DL_DIR", d, True), ud.type)
+        ud.viewdir          = os.path.join(ud.ccasedir, ud.viewname)
+        ud.configspecfile   = os.path.join(ud.ccasedir, ud.csname)
+        ud.localfile        = "%s.tar.gz" % (ud.identifier)
+
+        self.debug("host            = %s" % ud.host)
+        self.debug("path            = %s" % ud.path)
+        self.debug("server          = %s" % ud.server)
+        self.debug("proto           = %s" % ud.proto)
+        self.debug("type            = %s" % ud.type)
+        self.debug("vob             = %s" % ud.vob)
+        self.debug("module          = %s" % ud.module)
+        self.debug("basecmd         = %s" % ud.basecmd)
+        self.debug("label           = %s" % ud.label)
+        self.debug("ccasedir        = %s" % ud.ccasedir)
+        self.debug("viewdir         = %s" % ud.viewdir)
+        self.debug("viewname        = %s" % ud.viewname)
+        self.debug("configspecfile  = %s" % ud.configspecfile)
+        self.debug("localfile       = %s" % ud.localfile)
+
+        ud.localfile = os.path.join(data.getVar("DL_DIR", d, True), ud.localfile)
+
+    def _build_ccase_command(self, ud, command):
+        """
+        Build up a commandline based on ud
+        command is: mkview, setcs, rmview
+        """
+        options = []
+
+        if "rcleartool" in ud.basecmd:
+            options.append("-server %s" % ud.server)
+
+        basecmd = "%s %s" % (ud.basecmd, command)
+
+        if command is 'mkview':
+            if not "rcleartool" in ud.basecmd:
+                # Cleartool needs a -snapshot view
+                options.append("-snapshot")
+            options.append("-tag %s" % ud.viewname)
+            options.append(ud.viewdir)
+
+        elif command is 'rmview':
+            options.append("-force")
+            options.append("%s" % ud.viewdir)
+
+        elif command is 'setcs':
+            options.append("-overwrite")
+            options.append(ud.configspecfile)
+
+        else:
+            raise FetchError("Invalid ccase command %s" % command)
+
+        ccasecmd = "%s %s" % (basecmd, " ".join(options))
+        self.debug("ccasecmd = %s" % ccasecmd)
+        return ccasecmd
+
+    def _write_configspec(self, ud, d):
+        """
+        Create config spec file (ud.configspecfile) for ccase view
+        """
+        config_spec = ""
+        custom_config_spec = d.getVar("CCASE_CUSTOM_CONFIG_SPEC", d)
+        if custom_config_spec is not None:
+            for line in custom_config_spec.split("\\n"):
+                config_spec += line+"\n"
+            bb.warn("A custom config spec has been set, SRCREV is only relevant for the tarball name.")
+        else:
+            config_spec += "element * CHECKEDOUT\n"
+            config_spec += "element * %s\n" % ud.label
+            config_spec += "load %s%s\n" % (ud.vob, ud.module)
+
+        logger.info("Using config spec: \n%s" % config_spec)
+
+        with open(ud.configspecfile, 'w') as f:
+            f.write(config_spec)
+
+    def _remove_view(self, ud, d):
+        if os.path.exists(ud.viewdir):
+            os.chdir(ud.ccasedir)
+            cmd = self._build_ccase_command(ud, 'rmview');
+            logger.info("cleaning up [VOB=%s label=%s view=%s]", ud.vob, ud.label, ud.viewname)
+            bb.fetch2.check_network_access(d, cmd, ud.url)
+            output = runfetchcmd(cmd, d)
+            logger.info("rmview output: %s", output)
+
+    def need_update(self, ud, d):
+        if ("LATEST" in ud.label) or (ud.customspec and "LATEST" in ud.customspec):
+            ud.identifier += "-%s" % d.getVar("DATETIME",d, True)
+            return True
+        if os.path.exists(ud.localpath):
+            return False
+        return True
+
+    def supports_srcrev(self):
+        return True
+
+    def sortable_revision(self, ud, d, name):
+        return False, ud.identifier
+
+    def download(self, ud, d):
+        """Fetch url"""
+
+        # Make a fresh view
+        bb.utils.mkdirhier(ud.ccasedir)
+        self._write_configspec(ud, d)
+        cmd = self._build_ccase_command(ud, 'mkview')
+        logger.info("creating view [VOB=%s label=%s view=%s]", ud.vob, ud.label, ud.viewname)
+        bb.fetch2.check_network_access(d, cmd, ud.url)
+        try:
+            runfetchcmd(cmd, d)
+        except FetchError as e:
+            if "CRCLI2008E" in e.msg:
+                raise FetchError("%s\n%s\n" % (e.msg, "Call `rcleartool login` in your console to authenticate to the clearcase server before running bitbake."))
+            else:
+                raise e
+
+        # Set configspec: Setting the configspec effectively fetches the files as defined in the configspec
+        os.chdir(ud.viewdir)
+        cmd = self._build_ccase_command(ud, 'setcs');
+        logger.info("fetching data [VOB=%s label=%s view=%s]", ud.vob, ud.label, ud.viewname)
+        bb.fetch2.check_network_access(d, cmd, ud.url)
+        output = runfetchcmd(cmd, d)
+        logger.info("%s", output)
+
+        # Copy the configspec to the viewdir so we have it in our source tarball later
+        shutil.copyfile(ud.configspecfile, os.path.join(ud.viewdir, ud.csname))
+
+        # Clean clearcase meta-data before tar
+
+        runfetchcmd('tar -czf "%s" .' % (ud.localpath), d, cleanup = [ud.localpath])
+
+        # Clean up so we can create a new view next time
+        self.clean(ud, d);
+
+    def clean(self, ud, d):
+        self._remove_view(ud, d)
+        bb.utils.remove(ud.configspecfile)

bitbake/lib/bb/fetch2/cvs.py

@@ -0,0 +1,171 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Fetch' implementations
+
+Classes for obtaining upstream sources for the
+BitBake build tools.
+
+"""
+
+# Copyright (C) 2003, 2004  Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+#Based on functions from the base bb module, Copyright 2003 Holger Schurig
+#
+
+import os
+import logging
+import bb
+from bb.fetch2 import FetchMethod, FetchError, MissingParameterError, logger
+from bb.fetch2 import runfetchcmd
+
+class Cvs(FetchMethod):
+    """
+    Class to fetch a module or modules from cvs repositories
+    """
+    def supports(self, ud, d):
+        """
+        Check to see if a given url can be fetched with cvs.
+        """
+        return ud.type in ['cvs']
+
+    def urldata_init(self, ud, d):
+        if not "module" in ud.parm:
+            raise MissingParameterError("module", ud.url)
+        ud.module = ud.parm["module"]
+
+        ud.tag = ud.parm.get('tag', "")
+
+        # Override the default date in certain cases
+        if 'date' in ud.parm:
+            ud.date = ud.parm['date']
+        elif ud.tag:
+            ud.date = ""
+
+        norecurse = ''
+        if 'norecurse' in ud.parm:
+            norecurse = '_norecurse'
+
+        fullpath = ''
+        if 'fullpath' in ud.parm:
+            fullpath = '_fullpath'
+
+        ud.localfile = bb.data.expand('%s_%s_%s_%s%s%s.tar.gz' % (ud.module.replace('/', '.'), ud.host, ud.tag, ud.date, norecurse, fullpath), d)
+
+    def need_update(self, ud, d):
+        if (ud.date == "now"):
+            return True
+        if not os.path.exists(ud.localpath):
+            return True
+        return False
+
+    def download(self, ud, d):
+
+        method = ud.parm.get('method', 'pserver')
+        localdir = ud.parm.get('localdir', ud.module)
+        cvs_port = ud.parm.get('port', '')
+
+        cvs_rsh = None
+        if method == "ext":
+            if "rsh" in ud.parm:
+                cvs_rsh = ud.parm["rsh"]
+
+        if method == "dir":
+            cvsroot = ud.path
+        else:
+            cvsroot = ":" + method
+            cvsproxyhost = d.getVar('CVS_PROXY_HOST', True)
+            if cvsproxyhost:
+                cvsroot += ";proxy=" + cvsproxyhost
+            cvsproxyport = d.getVar('CVS_PROXY_PORT', True)
+            if cvsproxyport:
+                cvsroot += ";proxyport=" + cvsproxyport
+            cvsroot += ":" + ud.user
+            if ud.pswd:
+                cvsroot += ":" + ud.pswd
+            cvsroot += "@" + ud.host + ":" + cvs_port + ud.path
+
+        options = []
+        if 'norecurse' in ud.parm:
+            options.append("-l")
+        if ud.date:
+            # treat YYYYMMDDHHMM specially for CVS
+            if len(ud.date) == 12:
+                options.append("-D \"%s %s:%s UTC\"" % (ud.date[0:8], ud.date[8:10], ud.date[10:12]))
+            else:
+                options.append("-D \"%s UTC\"" % ud.date)
+        if ud.tag:
+            options.append("-r %s" % ud.tag)
+
+        cvsbasecmd = d.getVar("FETCHCMD_cvs", True)
+        cvscmd = cvsbasecmd + " '-d" + cvsroot + "' co " + " ".join(options) + " " + ud.module
+        cvsupdatecmd = cvsbasecmd + " '-d" + cvsroot + "' update -d -P " + " ".join(options)
+
+        if cvs_rsh:
+            cvscmd = "CVS_RSH=\"%s\" %s" % (cvs_rsh, cvscmd)
+            cvsupdatecmd = "CVS_RSH=\"%s\" %s" % (cvs_rsh, cvsupdatecmd)
+
+        # create module directory
+        logger.debug(2, "Fetch: checking for module directory")
+        pkg = d.getVar('PN', True)
+        pkgdir = os.path.join(d.getVar('CVSDIR', True), pkg)
+        moddir = os.path.join(pkgdir, localdir)
+        if os.access(os.path.join(moddir, 'CVS'), os.R_OK):
+            logger.info("Update " + ud.url)
+            bb.fetch2.check_network_access(d, cvsupdatecmd, ud.url)
+            # update sources there
+            os.chdir(moddir)
+            cmd = cvsupdatecmd
+        else:
+            logger.info("Fetch " + ud.url)
+            # check out sources there
+            bb.utils.mkdirhier(pkgdir)
+            os.chdir(pkgdir)
+            logger.debug(1, "Running %s", cvscmd)
+            bb.fetch2.check_network_access(d, cvscmd, ud.url)
+            cmd = cvscmd
+
+        runfetchcmd(cmd, d, cleanup = [moddir])
+
+        if not os.access(moddir, os.R_OK):
+            raise FetchError("Directory %s was not readable despite sucessful fetch?!" % moddir, ud.url)
+
+        scmdata = ud.parm.get("scmdata", "")
+        if scmdata == "keep":
+            tar_flags = ""
+        else:
+            tar_flags = "--exclude 'CVS'"
+
+        # tar them up to a defined filename
+        if 'fullpath' in ud.parm:
+            os.chdir(pkgdir)
+            cmd = "tar %s -czf %s %s" % (tar_flags, ud.localpath, localdir)
+        else:
+            os.chdir(moddir)
+            os.chdir('..')
+            cmd = "tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.basename(moddir))
+
+        runfetchcmd(cmd, d, cleanup = [ud.localpath])
+
+    def clean(self, ud, d):
+        """ Clean CVS Files and tarballs """
+
+        pkg = d.getVar('PN', True)
+        pkgdir = os.path.join(d.getVar("CVSDIR", True), pkg)
+
+        bb.utils.remove(pkgdir, True)
+        bb.utils.remove(ud.localpath)
+

bitbake/lib/bb/fetch2/git.py

@@ -0,0 +1,435 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Fetch' git implementation
+
+git fetcher support the SRC_URI with format of:
+SRC_URI = "git://some.host/somepath;OptionA=xxx;OptionB=xxx;..."
+
+Supported SRC_URI options are:
+
+- branch
+   The git branch to retrieve from. The default is "master"
+
+   This option also supports multiple branch fetching, with branches
+   separated by commas.  In multiple branches case, the name option
+   must have the same number of names to match the branches, which is
+   used to specify the SRC_REV for the branch
+   e.g:
+   SRC_URI="git://some.host/somepath;branch=branchX,branchY;name=nameX,nameY"
+   SRCREV_nameX = "xxxxxxxxxxxxxxxxxxxx"
+   SRCREV_nameY = "YYYYYYYYYYYYYYYYYYYY"
+
+- tag
+    The git tag to retrieve. The default is "master"
+
+- protocol
+   The method to use to access the repository. Common options are "git",
+   "http", "https", "file", "ssh" and "rsync". The default is "git".
+
+- rebaseable
+   rebaseable indicates that the upstream git repo may rebase in the future,
+   and current revision may disappear from upstream repo. This option will
+   remind fetcher to preserve local cache carefully for future use.
+   The default value is "0", set rebaseable=1 for rebaseable git repo.
+
+- nocheckout
+   Don't checkout source code when unpacking. set this option for the recipe
+   who has its own routine to checkout code.
+   The default is "0", set nocheckout=1 if needed.
+
+- bareclone
+   Create a bare clone of the source code and don't checkout the source code
+   when unpacking. Set this option for the recipe who has its own routine to
+   checkout code and tracking branch requirements.
+   The default is "0", set bareclone=1 if needed.
+
+- nobranch
+   Don't check the SHA validation for branch. set this option for the recipe
+   referring to commit which is valid in tag instead of branch.
+   The default is "0", set nobranch=1 if needed.
+
+"""
+
+#Copyright (C) 2005 Richard Purdie
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import errno
+import os
+import re
+import bb
+import errno
+from   bb    import data
+from   bb.fetch2 import FetchMethod
+from   bb.fetch2 import runfetchcmd
+from   bb.fetch2 import logger
+
+class Git(FetchMethod):
+    """Class to fetch a module or modules from git repositories"""
+    def init(self, d):
+        pass
+
+    def supports(self, ud, d):
+        """
+        Check to see if a given url can be fetched with git.
+        """
+        return ud.type in ['git']
+
+    def supports_checksum(self, urldata):
+        return False
+
+    def urldata_init(self, ud, d):
+        """
+        init git specific variable within url data
+        so that the git method like latest_revision() can work
+        """
+        if 'protocol' in ud.parm:
+            ud.proto = ud.parm['protocol']
+        elif not ud.host:
+            ud.proto = 'file'
+        else:
+            ud.proto = "git"
+
+        if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'):
+            raise bb.fetch2.ParameterError("Invalid protocol type", ud.url)
+
+        ud.nocheckout = ud.parm.get("nocheckout","0") == "1"
+
+        ud.rebaseable = ud.parm.get("rebaseable","0") == "1"
+
+        ud.nobranch = ud.parm.get("nobranch","0") == "1"
+
+        # bareclone implies nocheckout
+        ud.bareclone = ud.parm.get("bareclone","0") == "1"
+        if ud.bareclone:
+            ud.nocheckout = 1
+  
+        ud.unresolvedrev = {}
+        branches = ud.parm.get("branch", "master").split(',')
+        if len(branches) != len(ud.names):
+            raise bb.fetch2.ParameterError("The number of name and branch parameters is not balanced", ud.url)
+        ud.branches = {}
+        for name in ud.names:
+            branch = branches[ud.names.index(name)]
+            ud.branches[name] = branch
+            ud.unresolvedrev[name] = branch
+
+        ud.basecmd = data.getVar("FETCHCMD_git", d, True) or "git -c core.fsyncobjectfiles=0"
+
+        ud.write_tarballs = ((data.getVar("BB_GENERATE_MIRROR_TARBALLS", d, True) or "0") != "0") or ud.rebaseable
+
+        ud.setup_revisons(d)
+
+        for name in ud.names:
+            # Ensure anything that doesn't look like a sha256 checksum/revision is translated into one
+            if not ud.revisions[name] or len(ud.revisions[name]) != 40  or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]):
+                if ud.revisions[name]:
+                    ud.unresolvedrev[name] = ud.revisions[name]
+                ud.revisions[name] = self.latest_revision(ud, d, name)
+
+        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.'))
+        if gitsrcname.startswith('.'):
+            gitsrcname = gitsrcname[1:]
+
+        # for rebaseable git repo, it is necessary to keep mirror tar ball
+        # per revision, so that even the revision disappears from the
+        # upstream repo in the future, the mirror will remain intact and still
+        # contains the revision
+        if ud.rebaseable:
+            for name in ud.names:
+                gitsrcname = gitsrcname + '_' + ud.revisions[name]
+        ud.mirrortarball = 'git2_%s.tar.gz' % (gitsrcname)
+        ud.fullmirror = os.path.join(d.getVar("DL_DIR", True), ud.mirrortarball)
+        gitdir = d.getVar("GITDIR", True) or (d.getVar("DL_DIR", True) + "/git2/")
+        ud.clonedir = os.path.join(gitdir, gitsrcname)
+
+        ud.localfile = ud.clonedir
+
+    def localpath(self, ud, d):
+        return ud.clonedir
+
+    def need_update(self, ud, d):
+        if not os.path.exists(ud.clonedir):
+            return True
+        os.chdir(ud.clonedir)
+        for name in ud.names:
+            if not self._contains_ref(ud, d, name):
+                return True
+        if ud.write_tarballs and not os.path.exists(ud.fullmirror):
+            return True
+        return False
+
+    def try_premirror(self, ud, d):
+        # If we don't do this, updating an existing checkout with only premirrors
+        # is not possible
+        if d.getVar("BB_FETCH_PREMIRRORONLY", True) is not None:
+            return True
+        if os.path.exists(ud.clonedir):
+            return False
+        return True
+
+    def download(self, ud, d):
+        """Fetch url"""
+
+        # If the checkout doesn't exist and the mirror tarball does, extract it
+        if not os.path.exists(ud.clonedir) and os.path.exists(ud.fullmirror):
+            bb.utils.mkdirhier(ud.clonedir)
+            os.chdir(ud.clonedir)
+            runfetchcmd("tar -xzf %s" % (ud.fullmirror), d)
+
+        repourl = self._get_repo_url(ud)
+
+        # If the repo still doesn't exist, fallback to cloning it
+        if not os.path.exists(ud.clonedir):
+            # We do this since git will use a "-l" option automatically for local urls where possible
+            if repourl.startswith("file://"):
+                repourl = repourl[7:]
+            clone_cmd = "%s clone --bare --mirror %s %s" % (ud.basecmd, repourl, ud.clonedir)
+            if ud.proto.lower() != 'file':
+                bb.fetch2.check_network_access(d, clone_cmd)
+            runfetchcmd(clone_cmd, d)
+
+        os.chdir(ud.clonedir)
+        # Update the checkout if needed
+        needupdate = False
+        for name in ud.names:
+            if not self._contains_ref(ud, d, name):
+                needupdate = True
+        if needupdate:
+            try: 
+                runfetchcmd("%s remote rm origin" % ud.basecmd, d) 
+            except bb.fetch2.FetchError:
+                logger.debug(1, "No Origin")
+
+            runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, repourl), d)
+            fetch_cmd = "%s fetch -f --prune %s refs/*:refs/*" % (ud.basecmd, repourl)
+            if ud.proto.lower() != 'file':
+                bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
+            runfetchcmd(fetch_cmd, d)
+            runfetchcmd("%s prune-packed" % ud.basecmd, d)
+            runfetchcmd("%s pack-redundant --all | xargs -r rm" % ud.basecmd, d)
+            try:
+                os.unlink(ud.fullmirror)
+            except OSError as exc:
+                if exc.errno != errno.ENOENT:
+                    raise
+        os.chdir(ud.clonedir)
+        for name in ud.names:
+            if not self._contains_ref(ud, d, name):
+                raise bb.fetch2.FetchError("Unable to find revision %s in branch %s even from upstream" % (ud.revisions[name], ud.branches[name]))
+
+    def build_mirror_data(self, ud, d):
+        # Generate a mirror tarball if needed
+        if ud.write_tarballs and not os.path.exists(ud.fullmirror):
+            # it's possible that this symlink points to read-only filesystem with PREMIRROR
+            if os.path.islink(ud.fullmirror):
+                os.unlink(ud.fullmirror)
+
+            os.chdir(ud.clonedir)
+            logger.info("Creating tarball of git repository")
+            runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d)
+            runfetchcmd("touch %s.done" % (ud.fullmirror), d)
+
+    def unpack(self, ud, destdir, d):
+        """ unpack the downloaded src to destdir"""
+
+        subdir = ud.parm.get("subpath", "")
+        if subdir != "":
+            readpathspec = ":%s" % (subdir)
+            def_destsuffix = "%s/" % os.path.basename(subdir.rstrip('/'))
+        else:
+            readpathspec = ""
+            def_destsuffix = "git/"
+
+        destsuffix = ud.parm.get("destsuffix", def_destsuffix)
+        destdir = ud.destdir = os.path.join(destdir, destsuffix)
+        if os.path.exists(destdir):
+            bb.utils.prunedir(destdir)
+
+        cloneflags = "-s -n"
+        if ud.bareclone:
+            cloneflags += " --mirror"
+
+        runfetchcmd("%s clone %s %s/ %s" % (ud.basecmd, cloneflags, ud.clonedir, destdir), d)
+        os.chdir(destdir)
+        repourl = self._get_repo_url(ud)
+        runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, repourl), d)
+        if not ud.nocheckout:
+            if subdir != "":
+                runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.revisions[ud.names[0]], readpathspec), d)
+                runfetchcmd("%s checkout-index -q -f -a" % ud.basecmd, d)
+            elif not ud.nobranch:
+                branchname =  ud.branches[ud.names[0]]
+                runfetchcmd("%s checkout -B %s %s" % (ud.basecmd, branchname, \
+                            ud.revisions[ud.names[0]]), d)
+                runfetchcmd("%s branch --set-upstream %s origin/%s" % (ud.basecmd, branchname, \
+                            branchname), d)
+            else:
+                runfetchcmd("%s checkout %s" % (ud.basecmd, ud.revisions[ud.names[0]]), d)
+
+        return True
+
+    def clean(self, ud, d):
+        """ clean the git directory """
+
+        bb.utils.remove(ud.localpath, True)
+        bb.utils.remove(ud.fullmirror)
+        bb.utils.remove(ud.fullmirror + ".done")
+
+    def supports_srcrev(self):
+        return True
+
+    def _contains_ref(self, ud, d, name):
+        cmd = ""
+        if ud.nobranch:
+            cmd = "%s log --pretty=oneline -n 1 %s -- 2> /dev/null | wc -l" % (
+                ud.basecmd, ud.revisions[name])
+        else:
+            cmd =  "%s branch --contains %s --list %s 2> /dev/null | wc -l" % (
+                ud.basecmd, ud.revisions[name], ud.branches[name])
+        try:
+            output = runfetchcmd(cmd, d, quiet=True)
+        except bb.fetch2.FetchError:
+            return False
+        if len(output.split()) > 1:
+            raise bb.fetch2.FetchError("The command '%s' gave output with more then 1 line unexpectedly, output: '%s'" % (cmd, output))
+        return output.split()[0] != "0"
+
+    def _get_repo_url(self, ud):
+        """
+        Return the repository URL
+        """
+        if ud.user:
+            username = ud.user + '@'
+        else:
+            username = ""
+        return "%s://%s%s%s" % (ud.proto, username, ud.host, ud.path)
+
+    def _revision_key(self, ud, d, name):
+        """
+        Return a unique key for the url
+        """
+        return "git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name]
+
+    def _lsremote(self, ud, d, search):
+        """
+        Run git ls-remote with the specified search string
+        """
+        repourl = self._get_repo_url(ud)
+        cmd = "%s ls-remote %s %s" % \
+              (ud.basecmd, repourl, search)
+        if ud.proto.lower() != 'file':
+            bb.fetch2.check_network_access(d, cmd)
+        output = runfetchcmd(cmd, d, True)
+        if not output:
+            raise bb.fetch2.FetchError("The command %s gave empty output unexpectedly" % cmd, ud.url)
+        return output
+
+    def _latest_revision(self, ud, d, name):
+        """
+        Compute the HEAD revision for the url
+        """
+        output = self._lsremote(ud, d, "")
+        # Tags of the form ^{} may not work, need to fallback to other form
+        if ud.unresolvedrev[name][:5] == "refs/":
+            head = ud.unresolvedrev[name]
+            tag = ud.unresolvedrev[name]
+        else:
+            head = "refs/heads/%s" % ud.unresolvedrev[name]
+            tag = "refs/tags/%s" % ud.unresolvedrev[name]
+        for s in [head, tag + "^{}", tag]:
+            for l in output.split('\n'):
+                if s in l:
+                    return l.split()[0]
+        raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \
+            (ud.unresolvedrev[name], ud.host+ud.path))
+
+    def latest_versionstring(self, ud, d):
+        """
+        Compute the latest release name like "x.y.x" in "x.y.x+gitHASH"
+        by searching through the tags output of ls-remote, comparing
+        versions and returning the highest match.
+        """
+        pupver = ('', '')
+
+        tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX', True) or "(?P<pver>([0-9][\.|_]?)+)")
+        try:
+            output = self._lsremote(ud, d, "refs/tags/*")
+        except bb.fetch2.FetchError or bb.fetch2.NetworkAccess:
+            return pupver
+
+        verstring = ""
+        revision = ""
+        for line in output.split("\n"):
+            if not line:
+                break
+
+            tag_head = line.split("/")[-1]
+            # Ignore non-released branches
+            m = re.search("(alpha|beta|rc|final)+", tag_head)
+            if m:
+                continue
+
+            # search for version in the line
+            tag = tagregex.search(tag_head)
+            if tag == None:
+                continue
+
+            tag = tag.group('pver')
+            tag = tag.replace("_", ".")
+
+            if verstring and bb.utils.vercmp(("0", tag, ""), ("0", verstring, "")) < 0:
+                continue
+
+            verstring = tag
+            revision = line.split()[0]
+            pupver = (verstring, revision)
+
+        return pupver
+
+    def _build_revision(self, ud, d, name):
+        return ud.revisions[name]
+
+    def gitpkgv_revision(self, ud, d, name):
+        """
+        Return a sortable revision number by counting commits in the history
+        Based on gitpkgv.bblass in meta-openembedded
+        """
+        rev = self._build_revision(ud, d, name)
+        localpath = ud.localpath
+        rev_file = os.path.join(localpath, "oe-gitpkgv_" + rev)
+        if not os.path.exists(localpath):
+            commits = None
+        else:
+            if not os.path.exists(rev_file) or not os.path.getsize(rev_file):
+                from pipes import quote
+                commits = bb.fetch2.runfetchcmd(
+                        "git rev-list %s -- | wc -l" % (quote(rev)),
+                        d, quiet=True).strip().lstrip('0')
+                if commits:
+                    open(rev_file, "w").write("%d\n" % int(commits))
+            else:
+                commits = open(rev_file, "r").readline(128).strip()
+        if commits:
+            return False, "%s+%s" % (commits, rev[:7])
+        else:
+            return True, str(rev)
+
+    def checkstatus(self, fetch, ud, d):
+        try:
+            self._lsremote(ud, d, "")
+            return True
+        except FetchError:
+            return False

bitbake/lib/bb/fetch2/gitannex.py

@@ -0,0 +1,76 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Fetch' git annex implementation
+"""
+
+# Copyright (C) 2014 Otavio Salvador
+# Copyright (C) 2014 O.S. Systems Software LTDA.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import bb
+from   bb import data
+from   bb.fetch2.git import Git
+from   bb.fetch2 import runfetchcmd
+from   bb.fetch2 import logger
+
+class GitANNEX(Git):
+    def supports(self, ud, d):
+        """
+        Check to see if a given url can be fetched with git.
+        """
+        return ud.type in ['gitannex']
+
+    def uses_annex(self, ud, d):
+        for name in ud.names:
+            try:
+                runfetchcmd("%s rev-list git-annex" % (ud.basecmd), d, quiet=True)
+                return True
+            except bb.fetch.FetchError:
+                pass
+
+        return False
+
+    def update_annex(self, ud, d):
+        try:
+            runfetchcmd("%s annex get --all" % (ud.basecmd), d, quiet=True)
+        except bb.fetch.FetchError:
+            return False
+        runfetchcmd("chmod u+w -R %s/annex" % (ud.clonedir), d, quiet=True)
+
+        return True
+
+    def download(self, ud, d):
+        Git.download(self, ud, d)
+
+        os.chdir(ud.clonedir)
+        annex = self.uses_annex(ud, d)
+        if annex:
+            self.update_annex(ud, d)
+
+    def unpack(self, ud, destdir, d):
+        Git.unpack(self, ud, destdir, d)
+
+        os.chdir(ud.destdir)
+        try:
+            runfetchcmd("%s annex sync" % (ud.basecmd), d)
+        except bb.fetch.FetchError:
+            pass
+
+        annex = self.uses_annex(ud, d)
+        if annex:
+            runfetchcmd("%s annex get" % (ud.basecmd), d)
+            runfetchcmd("chmod u+w -R %s/.git/annex" % (ud.destdir), d, quiet=True)

bitbake/lib/bb/fetch2/gitsm.py

@@ -0,0 +1,134 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Fetch' git submodules implementation
+
+Inherits from and extends the Git fetcher to retrieve submodules of a git repository
+after cloning.
+
+SRC_URI = "gitsm://<see Git fetcher for syntax>"
+
+See the Git fetcher, git://, for usage documentation.
+
+NOTE: Switching a SRC_URI from "git://" to "gitsm://" requires a clean of your recipe.
+
+"""
+
+# Copyright (C) 2013 Richard Purdie
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import bb
+from   bb    import data
+from   bb.fetch2.git import Git
+from   bb.fetch2 import runfetchcmd
+from   bb.fetch2 import logger
+
+class GitSM(Git):
+    def supports(self, ud, d):
+        """
+        Check to see if a given url can be fetched with git.
+        """
+        return ud.type in ['gitsm']
+
+    def uses_submodules(self, ud, d):
+        for name in ud.names:
+            try:
+                runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revisions[name]), d, quiet=True)
+                return True
+            except bb.fetch.FetchError:
+                pass
+        return False
+
+    def _set_relative_paths(self, repopath):
+        """
+        Fix submodule paths to be relative instead of absolute,
+        so that when we move the repo it doesn't break
+        (In Git 1.7.10+ this is done automatically)
+        """
+        submodules = []
+        with open(os.path.join(repopath, '.gitmodules'), 'r') as f:
+            for line in f.readlines():
+                if line.startswith('[submodule'):
+                    submodules.append(line.split('"')[1])
+
+        for module in submodules:
+            repo_conf = os.path.join(repopath, module, '.git')
+            if os.path.exists(repo_conf):
+                with open(repo_conf, 'r') as f:
+                    lines = f.readlines()
+                newpath = ''
+                for i, line in enumerate(lines):
+                    if line.startswith('gitdir:'):
+                        oldpath = line.split(': ')[-1].rstrip()
+                        if oldpath.startswith('/'):
+                            newpath = '../' * (module.count('/') + 1) + '.git/modules/' + module
+                            lines[i] = 'gitdir: %s\n' % newpath
+                            break
+                if newpath:
+                    with open(repo_conf, 'w') as f:
+                        for line in lines:
+                            f.write(line)
+
+            repo_conf2 = os.path.join(repopath, '.git', 'modules', module, 'config')
+            if os.path.exists(repo_conf2):
+                with open(repo_conf2, 'r') as f:
+                    lines = f.readlines()
+                newpath = ''
+                for i, line in enumerate(lines):
+                    if line.lstrip().startswith('worktree = '):
+                        oldpath = line.split(' = ')[-1].rstrip()
+                        if oldpath.startswith('/'):
+                            newpath = '../' * (module.count('/') + 3) + module
+                            lines[i] = '\tworktree = %s\n' % newpath
+                            break
+                if newpath:
+                    with open(repo_conf2, 'w') as f:
+                        for line in lines:
+                            f.write(line)
+
+    def update_submodules(self, ud, d):
+        # We have to convert bare -> full repo, do the submodule bit, then convert back
+        tmpclonedir = ud.clonedir + ".tmp"
+        gitdir = tmpclonedir + os.sep + ".git"
+        bb.utils.remove(tmpclonedir, True)
+        os.mkdir(tmpclonedir)
+        os.rename(ud.clonedir, gitdir)
+        runfetchcmd("sed " + gitdir + "/config -i -e 's/bare.*=.*true/bare = false/'", d)
+        os.chdir(tmpclonedir)
+        runfetchcmd(ud.basecmd + " reset --hard", d)
+        runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d)
+        runfetchcmd(ud.basecmd + " submodule update --init --recursive", d)
+        self._set_relative_paths(tmpclonedir)
+        runfetchcmd("sed " + gitdir + "/config -i -e 's/bare.*=.*false/bare = true/'", d)
+        os.rename(gitdir, ud.clonedir,)
+        bb.utils.remove(tmpclonedir, True)
+
+    def download(self, ud, d):
+        Git.download(self, ud, d)
+
+        os.chdir(ud.clonedir)
+        submodules = self.uses_submodules(ud, d)
+        if submodules:
+            self.update_submodules(ud, d)
+
+    def unpack(self, ud, destdir, d):
+        Git.unpack(self, ud, destdir, d)
+        
+        os.chdir(ud.destdir)
+        submodules = self.uses_submodules(ud, d)
+        if submodules:
+            runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d)
+            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d)

bitbake/lib/bb/fetch2/hg.py

@@ -0,0 +1,278 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Fetch' implementation for mercurial DRCS (hg).
+
+"""
+
+# Copyright (C) 2003, 2004  Chris Larson
+# Copyright (C) 2004        Marcin Juszkiewicz
+# Copyright (C) 2007        Robert Schuster
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+import os
+import sys
+import logging
+import bb
+import errno
+from bb import data
+from bb.fetch2 import FetchMethod
+from bb.fetch2 import FetchError
+from bb.fetch2 import MissingParameterError
+from bb.fetch2 import runfetchcmd
+from bb.fetch2 import logger
+
+class Hg(FetchMethod):
+    """Class to fetch from mercurial repositories"""
+    def supports(self, ud, d):
+        """
+        Check to see if a given url can be fetched with mercurial.
+        """
+        return ud.type in ['hg']
+
+    def supports_checksum(self, urldata):
+        """
+        Don't require checksums for local archives created from
+        repository checkouts.
+        """ 
+        return False
+
+    def urldata_init(self, ud, d):
+        """
+        init hg specific variable within url data
+        """
+        if not "module" in ud.parm:
+            raise MissingParameterError('module', ud.url)
+
+        ud.module = ud.parm["module"]
+
+        if 'protocol' in ud.parm:
+            ud.proto = ud.parm['protocol']
+        elif not ud.host:
+            ud.proto = 'file'
+        else:
+            ud.proto = "hg"
+
+        ud.setup_revisons(d)
+
+        if 'rev' in ud.parm:
+            ud.revision = ud.parm['rev']
+        elif not ud.revision:
+            ud.revision = self.latest_revision(ud, d)
+
+        # Create paths to mercurial checkouts
+        hgsrcname = '%s_%s_%s' % (ud.module.replace('/', '.'), \
+                            ud.host, ud.path.replace('/', '.'))
+        ud.mirrortarball = 'hg_%s.tar.gz' % hgsrcname
+        ud.fullmirror = os.path.join(d.getVar("DL_DIR", True), ud.mirrortarball)
+
+        hgdir = d.getVar("HGDIR", True) or (d.getVar("DL_DIR", True) + "/hg/")
+        ud.pkgdir = os.path.join(hgdir, hgsrcname)
+        ud.moddir = os.path.join(ud.pkgdir, ud.module)
+        ud.localfile = ud.moddir
+        ud.basecmd = data.getVar("FETCHCMD_hg", d, True) or "/usr/bin/env hg"
+
+        ud.write_tarballs = d.getVar("BB_GENERATE_MIRROR_TARBALLS", True)
+
+    def need_update(self, ud, d):
+        revTag = ud.parm.get('rev', 'tip')
+        if revTag == "tip":
+            return True
+        if not os.path.exists(ud.localpath):
+            return True
+        return False
+
+    def try_premirror(self, ud, d):
+        # If we don't do this, updating an existing checkout with only premirrors
+        # is not possible
+        if d.getVar("BB_FETCH_PREMIRRORONLY", True) is not None:
+            return True
+        if os.path.exists(ud.moddir):
+            return False
+        return True
+
+    def _buildhgcommand(self, ud, d, command):
+        """
+        Build up an hg commandline based on ud
+        command is "fetch", "update", "info"
+        """
+
+        proto = ud.parm.get('protocol', 'http')
+
+        host = ud.host
+        if proto == "file":
+            host = "/"
+            ud.host = "localhost"
+
+        if not ud.user:
+            hgroot = host + ud.path
+        else:
+            if ud.pswd:
+                hgroot = ud.user + ":" + ud.pswd + "@" + host + ud.path
+            else:
+                hgroot = ud.user + "@" + host + ud.path
+
+        if command == "info":
+            return "%s identify -i %s://%s/%s" % (ud.basecmd, proto, hgroot, ud.module)
+
+        options = [];
+
+        # Don't specify revision for the fetch; clone the entire repo.
+        # This avoids an issue if the specified revision is a tag, because
+        # the tag actually exists in the specified revision + 1, so it won't
+        # be available when used in any successive commands.
+        if ud.revision and command != "fetch":
+            options.append("-r %s" % ud.revision)
+
+        if command == "fetch":
+            if ud.user and ud.pswd:
+                cmd = "%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" clone %s %s://%s/%s %s" % (ud.basecmd, ud.user, ud.pswd, proto, " ".join(options), proto, hgroot, ud.module, ud.module)
+            else:
+                cmd = "%s clone %s %s://%s/%s %s" % (ud.basecmd, " ".join(options), proto, hgroot, ud.module, ud.module)
+        elif command == "pull":
+            # do not pass options list; limiting pull to rev causes the local
+            # repo not to contain it and immediately following "update" command
+            # will crash
+            if ud.user and ud.pswd:
+                cmd = "%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" pull" % (ud.basecmd, ud.user, ud.pswd, proto)
+            else:
+                cmd = "%s pull" % (ud.basecmd)
+        elif command == "update":
+            if ud.user and ud.pswd:
+                cmd = "%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" update -C %s" % (ud.basecmd, ud.user, ud.pswd, proto, " ".join(options))
+            else:
+                cmd = "%s update -C %s" % (ud.basecmd, " ".join(options))
+        else:
+            raise FetchError("Invalid hg command %s" % command, ud.url)
+
+        return cmd
+
+    def download(self, ud, d):
+        """Fetch url"""
+
+        logger.debug(2, "Fetch: checking for module directory '" + ud.moddir + "'")
+
+        # If the checkout doesn't exist and the mirror tarball does, extract it
+        if not os.path.exists(ud.pkgdir) and os.path.exists(ud.fullmirror):
+            bb.utils.mkdirhier(ud.pkgdir)
+            os.chdir(ud.pkgdir)
+            runfetchcmd("tar -xzf %s" % (ud.fullmirror), d)
+
+        if os.access(os.path.join(ud.moddir, '.hg'), os.R_OK):
+            # Found the source, check whether need pull
+            updatecmd = self._buildhgcommand(ud, d, "update")
+            os.chdir(ud.moddir)
+            logger.debug(1, "Running %s", updatecmd)
+            try:
+                runfetchcmd(updatecmd, d)
+            except bb.fetch2.FetchError:
+                # Runnning pull in the repo
+                pullcmd = self._buildhgcommand(ud, d, "pull")
+                logger.info("Pulling " + ud.url)
+                # update sources there
+                os.chdir(ud.moddir)
+                logger.debug(1, "Running %s", pullcmd)
+                bb.fetch2.check_network_access(d, pullcmd, ud.url)
+                runfetchcmd(pullcmd, d)
+                try:
+                    os.unlink(ud.fullmirror)
+                except OSError as exc:
+                    if exc.errno != errno.ENOENT:
+                        raise
+
+        # No source found, clone it.
+        if not os.path.exists(ud.moddir):
+            fetchcmd = self._buildhgcommand(ud, d, "fetch")
+            logger.info("Fetch " + ud.url)
+            # check out sources there
+            bb.utils.mkdirhier(ud.pkgdir)
+            os.chdir(ud.pkgdir)
+            logger.debug(1, "Running %s", fetchcmd)
+            bb.fetch2.check_network_access(d, fetchcmd, ud.url)
+            runfetchcmd(fetchcmd, d)
+
+        # Even when we clone (fetch), we still need to update as hg's clone
+        # won't checkout the specified revision if its on a branch
+        updatecmd = self._buildhgcommand(ud, d, "update")
+        os.chdir(ud.moddir)
+        logger.debug(1, "Running %s", updatecmd)
+        runfetchcmd(updatecmd, d)
+
+    def clean(self, ud, d):
+        """ Clean the hg dir """
+
+        bb.utils.remove(ud.localpath, True)
+        bb.utils.remove(ud.fullmirror)
+        bb.utils.remove(ud.fullmirror + ".done")
+
+    def supports_srcrev(self):
+        return True
+
+    def _latest_revision(self, ud, d, name):
+        """
+        Compute tip revision for the url
+        """
+        bb.fetch2.check_network_access(d, self._buildhgcommand(ud, d, "info"))
+        output = runfetchcmd(self._buildhgcommand(ud, d, "info"), d)
+        return output.strip()
+
+    def _build_revision(self, ud, d, name):
+        return ud.revision
+
+    def _revision_key(self, ud, d, name):
+        """
+        Return a unique key for the url
+        """
+        return "hg:" + ud.moddir
+
+    def build_mirror_data(self, ud, d):
+        # Generate a mirror tarball if needed
+        if ud.write_tarballs == "1" and not os.path.exists(ud.fullmirror):
+            # it's possible that this symlink points to read-only filesystem with PREMIRROR
+            if os.path.islink(ud.fullmirror):
+                os.unlink(ud.fullmirror)
+
+            os.chdir(ud.pkgdir)
+            logger.info("Creating tarball of hg repository")
+            runfetchcmd("tar -czf %s %s" % (ud.fullmirror, ud.module), d)
+            runfetchcmd("touch %s.done" % (ud.fullmirror), d)
+
+    def localpath(self, ud, d):
+        return ud.pkgdir
+
+    def unpack(self, ud, destdir, d):
+        """
+        Make a local clone or export for the url
+        """
+
+        revflag = "-r %s" % ud.revision
+        subdir = ud.parm.get("destsuffix", ud.module)
+        codir = "%s/%s" % (destdir, subdir)
+
+        scmdata = ud.parm.get("scmdata", "")
+        if scmdata != "nokeep":
+            if not os.access(os.path.join(codir, '.hg'), os.R_OK):
+                logger.debug(2, "Unpack: creating new hg repository in '" + codir + "'")
+                runfetchcmd("%s init %s" % (ud.basecmd, codir), d)
+            logger.debug(2, "Unpack: updating source in '" + codir + "'")
+            os.chdir(codir)
+            runfetchcmd("%s pull %s" % (ud.basecmd, ud.moddir), d)
+            runfetchcmd("%s up -C %s" % (ud.basecmd, revflag), d)
+        else:
+            logger.debug(2, "Unpack: extracting source to '" + codir + "'")
+            os.chdir(ud.moddir)
+            runfetchcmd("%s archive -t files %s %s" % (ud.basecmd, revflag, codir), d)

bitbake/lib/bb/fetch2/local.py

@@ -0,0 +1,129 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Fetch' implementations
+
+Classes for obtaining upstream sources for the
+BitBake build tools.
+
+"""
+
+# Copyright (C) 2003, 2004  Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+import os
+import urllib
+import bb
+import bb.utils
+from   bb import data
+from   bb.fetch2 import FetchMethod, FetchError
+from   bb.fetch2 import logger
+
+class Local(FetchMethod):
+    def supports(self, urldata, d):
+        """
+        Check to see if a given url represents a local fetch.
+        """
+        return urldata.type in ['file']
+
+    def urldata_init(self, ud, d):
+        # We don't set localfile as for this fetcher the file is already local!
+        ud.decodedurl = urllib.unquote(ud.url.split("://")[1].split(";")[0])
+        ud.basename = os.path.basename(ud.decodedurl)
+        ud.basepath = ud.decodedurl
+        ud.needdonestamp = False
+        return
+
+    def localpath(self, urldata, d):
+        """
+        Return the local filename of a given url assuming a successful fetch.
+        """
+        return self.localpaths(urldata, d)[-1]
+
+    def localpaths(self, urldata, d):
+        """
+        Return the local filename of a given url assuming a successful fetch.
+        """
+        searched = []
+        path = urldata.decodedurl
+        newpath = path
+        if path[0] == "/":
+            return [path]
+        filespath = data.getVar('FILESPATH', d, True)
+        if filespath:
+            logger.debug(2, "Searching for %s in paths:\n    %s" % (path, "\n    ".join(filespath.split(":"))))
+            newpath, hist = bb.utils.which(filespath, path, history=True)
+            searched.extend(hist)
+        if not newpath:
+            filesdir = data.getVar('FILESDIR', d, True)
+            if filesdir:
+                logger.debug(2, "Searching for %s in path: %s" % (path, filesdir))
+                newpath = os.path.join(filesdir, path)
+                searched.append(newpath)
+        if (not newpath or not os.path.exists(newpath)) and path.find("*") != -1:
+            # For expressions using '*', best we can do is take the first directory in FILESPATH that exists
+            newpath, hist = bb.utils.which(filespath, ".", history=True)
+            searched.extend(hist)
+            logger.debug(2, "Searching for %s in path: %s" % (path, newpath))
+            return searched
+        if not os.path.exists(newpath):
+            dldirfile = os.path.join(d.getVar("DL_DIR", True), path)
+            logger.debug(2, "Defaulting to %s for %s" % (dldirfile, path))
+            bb.utils.mkdirhier(os.path.dirname(dldirfile))
+            searched.append(dldirfile)
+            return searched
+        return searched
+
+    def need_update(self, ud, d):
+        if ud.url.find("*") != -1:
+            return False
+        if os.path.exists(ud.localpath):
+            return False
+        return True
+
+    def download(self, urldata, d):
+        """Fetch urls (no-op for Local method)"""
+        # no need to fetch local files, we'll deal with them in place.
+        if self.supports_checksum(urldata) and not os.path.exists(urldata.localpath):
+            locations = []
+            filespath = data.getVar('FILESPATH', d, True)
+            if filespath:
+                locations = filespath.split(":")
+            filesdir = data.getVar('FILESDIR', d, True)
+            if filesdir:
+                locations.append(filesdir)
+            locations.append(d.getVar("DL_DIR", True))
+
+            msg = "Unable to find file " + urldata.url + " anywhere. The paths that were searched were:\n    " + "\n    ".join(locations)
+            raise FetchError(msg)
+
+        return True
+
+    def checkstatus(self, fetch, urldata, d):
+        """
+        Check the status of the url
+        """
+        if urldata.localpath.find("*") != -1:
+            logger.info("URL %s looks like a glob and was therefore not checked.", urldata.url)
+            return True
+        if os.path.exists(urldata.localpath):
+            return True
+        return False
+
+    def clean(self, urldata, d):
+        return
+

bitbake/lib/bb/fetch2/npm.py

@@ -0,0 +1,284 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Fetch' NPM implementation
+
+The NPM fetcher is used to retrieve files from the npmjs repository
+
+Usage in the recipe:
+
+    SRC_URI = "npm://registry.npmjs.org/;name=${PN};version=${PV}"
+    Suported SRC_URI options are:
+
+    - name
+    - version
+
+    npm://registry.npmjs.org/${PN}/-/${PN}-${PV}.tgz  would become npm://registry.npmjs.org;name=${PN};ver=${PV}
+    The fetcher all triggers off the existence of ud.localpath. If that exists and has the ".done" stamp, its assumed the fetch is good/done
+
+"""
+
+import os
+import sys
+import urllib
+import json
+import subprocess
+import signal
+import bb
+from   bb import data
+from   bb.fetch2 import FetchMethod
+from   bb.fetch2 import FetchError
+from   bb.fetch2 import ChecksumError
+from   bb.fetch2 import runfetchcmd
+from   bb.fetch2 import logger
+from   bb.fetch2 import UnpackError
+from   bb.fetch2 import ParameterError
+from   distutils import spawn
+
+def subprocess_setup():
+    # Python installs a SIGPIPE handler by default. This is usually not what
+    # non-Python subprocesses expect.
+    # SIGPIPE errors are known issues with gzip/bash
+    signal.signal(signal.SIGPIPE, signal.SIG_DFL)
+
+class Npm(FetchMethod):
+
+    """Class to fetch urls via 'npm'"""
+    def init(self, d):
+        pass
+
+    def supports(self, ud, d):
+        """
+        Check to see if a given url can be fetched with npm
+        """
+        return ud.type in ['npm']
+
+    def debug(self, msg):
+        logger.debug(1, "NpmFetch: %s", msg)
+
+    def clean(self, ud, d):
+        logger.debug(2, "Calling cleanup %s" % ud.pkgname)
+        bb.utils.remove(ud.localpath, False)
+        bb.utils.remove(ud.pkgdatadir, True)
+        bb.utils.remove(ud.fullmirror, False)
+
+    def urldata_init(self, ud, d):
+        """
+        init NPM specific variable within url data
+        """
+        if 'downloadfilename' in ud.parm:
+            ud.basename = ud.parm['downloadfilename']
+        else:
+            ud.basename = os.path.basename(ud.path)
+
+        # can't call it ud.name otherwise fetcher base class will start doing sha1stuff
+        # TODO: find a way to get an sha1/sha256 manifest of pkg & all deps
+        ud.pkgname = ud.parm.get("name", None)
+        if not ud.pkgname:
+            raise ParameterError("NPM fetcher requires a name parameter", ud.url)
+        ud.version = ud.parm.get("version", None)
+        if not ud.version:
+            raise ParameterError("NPM fetcher requires a version parameter", ud.url)
+        ud.bbnpmmanifest = "%s-%s.deps.json" % (ud.pkgname, ud.version)
+        ud.registry = "http://%s" % (ud.url.replace('npm://', '', 1).split(';'))[0]
+        prefixdir = "npm/%s" % ud.pkgname
+        ud.pkgdatadir = d.expand("${DL_DIR}/%s" % prefixdir)
+        if not os.path.exists(ud.pkgdatadir):
+            bb.utils.mkdirhier(ud.pkgdatadir)
+        ud.localpath = d.expand("${DL_DIR}/npm/%s" % ud.bbnpmmanifest)
+
+        self.basecmd = d.getVar("FETCHCMD_wget", True) or "/usr/bin/env wget -O -t 2 -T 30 -nv --passive-ftp --no-check-certificate "
+        self.basecmd += " --directory-prefix=%s " % prefixdir
+
+        ud.write_tarballs = ((data.getVar("BB_GENERATE_MIRROR_TARBALLS", d, True) or "0") != "0")
+        ud.mirrortarball = 'npm_%s-%s.tar.xz' % (ud.pkgname, ud.version)
+        ud.fullmirror = os.path.join(d.getVar("DL_DIR", True), ud.mirrortarball)
+
+    def need_update(self, ud, d):
+        if os.path.exists(ud.localpath):
+            return False
+        return True
+
+    def _runwget(self, ud, d, command, quiet):
+        logger.debug(2, "Fetching %s using command '%s'" % (ud.url, command))
+        bb.fetch2.check_network_access(d, command)
+        runfetchcmd(command, d, quiet)
+
+    def _unpackdep(self, ud, pkg, data, destdir, dldir, d):
+        file = data[pkg]['tgz']
+        logger.debug(2, "file to extract is %s" % file)
+        if file.endswith('.tgz') or file.endswith('.tar.gz') or file.endswith('.tar.Z'):
+            cmd = 'tar xz --strip 1 --no-same-owner --warning=no-unknown-keyword -f %s/%s' % (dldir, file)
+        else:
+            bb.fatal("NPM package %s downloaded not a tarball!" % file)
+
+        # Change to subdir before executing command
+        save_cwd = os.getcwd()
+        if not os.path.exists(destdir):
+            os.makedirs(destdir)
+        os.chdir(destdir)
+        path = d.getVar('PATH', True)
+        if path:
+            cmd = "PATH=\"%s\" %s" % (path, cmd)
+        bb.note("Unpacking %s to %s/" % (file, os.getcwd()))
+        ret = subprocess.call(cmd, preexec_fn=subprocess_setup, shell=True)
+        os.chdir(save_cwd)
+
+        if ret != 0:
+            raise UnpackError("Unpack command %s failed with return value %s" % (cmd, ret), ud.url)
+
+        if 'deps' not in data[pkg]:
+            return
+        for dep in data[pkg]['deps']:
+            self._unpackdep(ud, dep, data[pkg]['deps'], "%s/node_modules/%s" % (destdir, dep), dldir, d)
+
+
+    def unpack(self, ud, destdir, d):
+        dldir = d.getVar("DL_DIR", True)
+        depdumpfile = "%s-%s.deps.json" % (ud.pkgname, ud.version)
+        with open("%s/npm/%s" % (dldir, depdumpfile)) as datafile:
+            workobj = json.load(datafile)
+        dldir = "%s/%s" % (os.path.dirname(ud.localpath), ud.pkgname)
+
+        self._unpackdep(ud, ud.pkgname, workobj,  "%s/npmpkg" % destdir, dldir, d)
+
+    def _parse_view(self, output):
+        '''
+        Parse the output of npm view --json; the last JSON result
+        is assumed to be the one that we're interested in.
+        '''
+        pdata = None
+        outdeps = {}
+        datalines = []
+        bracelevel = 0
+        for line in output.splitlines():
+            if bracelevel:
+                datalines.append(line)
+            elif '{' in line:
+                datalines = []
+                datalines.append(line)
+            bracelevel = bracelevel + line.count('{') - line.count('}')
+        if datalines:
+            pdata = json.loads('\n'.join(datalines))
+        return pdata
+
+    def _getdependencies(self, pkg, data, version, d, ud, optional=False):
+        pkgfullname = pkg
+        if version != '*' and not '/' in version:
+            pkgfullname += "@'%s'" % version
+        logger.debug(2, "Calling getdeps on %s" % pkg)
+        fetchcmd = "npm view %s --json --registry %s" % (pkgfullname, ud.registry)
+        output = runfetchcmd(fetchcmd, d, True)
+        pdata = self._parse_view(output)
+        if not pdata:
+            raise FetchError("The command '%s' returned no output" % fetchcmd)
+        if optional:
+            pkg_os = pdata.get('os', None)
+            if pkg_os:
+                if not isinstance(pkg_os, list):
+                    pkg_os = [pkg_os]
+                if 'linux' not in pkg_os or '!linux' in pkg_os:
+                    logger.debug(2, "Skipping %s since it's incompatible with Linux" % pkg)
+                    return
+        #logger.debug(2, "Output URL is %s - %s - %s" % (ud.basepath, ud.basename, ud.localfile))
+        outputurl = pdata['dist']['tarball']
+        data[pkg] = {}
+        data[pkg]['tgz'] = os.path.basename(outputurl)
+        self._runwget(ud, d, "%s %s" % (self.basecmd, outputurl), False)
+
+        dependencies = pdata.get('dependencies', {})
+        optionalDependencies = pdata.get('optionalDependencies', {})
+        depsfound = {}
+        optdepsfound = {}
+        data[pkg]['deps'] = {}
+        for dep in dependencies:
+            if dep in optionalDependencies:
+                optdepsfound[dep] = dependencies[dep]
+            else:
+                depsfound[dep] = dependencies[dep]
+        for dep, version in optdepsfound.iteritems():
+            self._getdependencies(dep, data[pkg]['deps'], version, d, ud, optional=True)
+        for dep, version in depsfound.iteritems():
+            self._getdependencies(dep, data[pkg]['deps'], version, d, ud)
+
+    def _getshrinkeddependencies(self, pkg, data, version, d, ud, lockdown, manifest):
+        logger.debug(2, "NPM shrinkwrap file is %s" % data)
+        outputurl = "invalid"
+        if ('resolved' not in data) or (not data['resolved'].startswith('http')):
+            # will be the case for ${PN}
+            fetchcmd = "npm view %s@%s dist.tarball --registry %s" % (pkg, version, ud.registry)
+            logger.debug(2, "Found this matching URL: %s" % str(fetchcmd))
+            outputurl = runfetchcmd(fetchcmd, d, True)
+        else:
+            outputurl = data['resolved']
+        self._runwget(ud, d, "%s %s" % (self.basecmd, outputurl), False)
+        manifest[pkg] = {}
+        manifest[pkg]['tgz'] = os.path.basename(outputurl).rstrip()
+        manifest[pkg]['deps'] = {}
+
+        if pkg in lockdown:
+            sha1_expected = lockdown[pkg][version]
+            sha1_data = bb.utils.sha1_file("npm/%s/%s" % (ud.pkgname, manifest[pkg]['tgz']))
+            if sha1_expected != sha1_data:
+                msg = "\nFile: '%s' has %s checksum %s when %s was expected" % (manifest[pkg]['tgz'], 'sha1', sha1_data, sha1_expected)
+                raise ChecksumError('Checksum mismatch!%s' % msg)
+        else:
+            logger.debug(2, "No lockdown data for %s@%s" % (pkg, version))
+
+        if 'dependencies' in data:
+            for obj in data['dependencies']:
+                logger.debug(2, "Found dep is %s" % str(obj))
+                self._getshrinkeddependencies(obj, data['dependencies'][obj], data['dependencies'][obj]['version'], d, ud, lockdown, manifest[pkg]['deps'])
+
+    def download(self, ud, d):
+        """Fetch url"""
+        jsondepobj = {}
+        shrinkobj = {}
+        lockdown = {}
+
+        if not os.listdir(ud.pkgdatadir) and os.path.exists(ud.fullmirror):
+            dest = d.getVar("DL_DIR", True)
+            bb.utils.mkdirhier(dest)
+            save_cwd = os.getcwd()
+            os.chdir(dest)
+            runfetchcmd("tar -xJf %s" % (ud.fullmirror), d)
+            os.chdir(save_cwd)
+            return
+
+        shwrf = d.getVar('NPM_SHRINKWRAP', True)
+        logger.debug(2, "NPM shrinkwrap file is %s" % shwrf)
+        try:
+            with open(shwrf) as datafile:
+                shrinkobj = json.load(datafile)
+        except:
+            logger.warn('Missing shrinkwrap file in NPM_SHRINKWRAP for %s, this will lead to unreliable builds!' % ud.pkgname)
+        lckdf = d.getVar('NPM_LOCKDOWN', True)
+        logger.debug(2, "NPM lockdown file is %s" % lckdf)
+        try:
+            with open(lckdf) as datafile:
+                lockdown = json.load(datafile)
+        except:
+            logger.warn('Missing lockdown file in NPM_LOCKDOWN for %s, this will lead to unreproducible builds!' % ud.pkgname)
+
+        if ('name' not in shrinkobj):
+            self._getdependencies(ud.pkgname, jsondepobj, ud.version, d, ud)
+        else:
+            self._getshrinkeddependencies(ud.pkgname, shrinkobj, ud.version, d, ud, lockdown, jsondepobj)
+
+        with open(ud.localpath, 'w') as outfile:
+            json.dump(jsondepobj, outfile)
+
+    def build_mirror_data(self, ud, d):
+        # Generate a mirror tarball if needed
+        if ud.write_tarballs and not os.path.exists(ud.fullmirror):
+            # it's possible that this symlink points to read-only filesystem with PREMIRROR
+            if os.path.islink(ud.fullmirror):
+                os.unlink(ud.fullmirror)
+
+            save_cwd = os.getcwd()
+            os.chdir(d.getVar("DL_DIR", True))
+            logger.info("Creating tarball of npm data")
+            runfetchcmd("tar -cJf %s npm/%s npm/%s" % (ud.fullmirror, ud.bbnpmmanifest, ud.pkgname), d)
+            runfetchcmd("touch %s.done" % (ud.fullmirror), d)
+            os.chdir(save_cwd)
+

bitbake/lib/bb/fetch2/osc.py

@@ -0,0 +1,135 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+Bitbake "Fetch" implementation for osc (Opensuse build service client).
+Based on the svn "Fetch" implementation.
+
+"""
+
+import  os
+import  sys
+import logging
+import  bb
+from    bb       import data
+from    bb.fetch2 import FetchMethod
+from    bb.fetch2 import FetchError
+from    bb.fetch2 import MissingParameterError
+from    bb.fetch2 import runfetchcmd
+
+class Osc(FetchMethod):
+    """Class to fetch a module or modules from Opensuse build server
+       repositories."""
+
+    def supports(self, ud, d):
+        """
+        Check to see if a given url can be fetched with osc.
+        """
+        return ud.type in ['osc']
+
+    def urldata_init(self, ud, d):
+        if not "module" in ud.parm:
+            raise MissingParameterError('module', ud.url)
+
+        ud.module = ud.parm["module"]
+
+        # Create paths to osc checkouts
+        relpath = self._strip_leading_slashes(ud.path)
+        ud.pkgdir = os.path.join(d.getVar('OSCDIR', True), ud.host)
+        ud.moddir = os.path.join(ud.pkgdir, relpath, ud.module)
+
+        if 'rev' in ud.parm:
+            ud.revision = ud.parm['rev']
+        else:
+            pv = d.getVar("PV", False)
+            rev = bb.fetch2.srcrev_internal_helper(ud, d)
+            if rev and rev != True:
+                ud.revision = rev
+            else:
+                ud.revision = ""
+
+        ud.localfile = data.expand('%s_%s_%s.tar.gz' % (ud.module.replace('/', '.'), ud.path.replace('/', '.'), ud.revision), d)
+
+    def _buildosccommand(self, ud, d, command):
+        """
+        Build up an ocs commandline based on ud
+        command is "fetch", "update", "info"
+        """
+
+        basecmd = data.expand('${FETCHCMD_osc}', d)
+
+        proto = ud.parm.get('protocol', 'ocs')
+
+        options = []
+
+        config = "-c %s" % self.generate_config(ud, d)
+
+        if ud.revision:
+            options.append("-r %s" % ud.revision)
+
+        coroot = self._strip_leading_slashes(ud.path)
+
+        if command == "fetch":
+            osccmd = "%s %s co %s/%s %s" % (basecmd, config, coroot, ud.module, " ".join(options))
+        elif command == "update":
+            osccmd = "%s %s up %s" % (basecmd, config, " ".join(options))
+        else:
+            raise FetchError("Invalid osc command %s" % command, ud.url)
+
+        return osccmd
+
+    def download(self, ud, d):
+        """
+        Fetch url
+        """
+
+        logger.debug(2, "Fetch: checking for module directory '" + ud.moddir + "'")
+
+        if os.access(os.path.join(d.getVar('OSCDIR', True), ud.path, ud.module), os.R_OK):
+            oscupdatecmd = self._buildosccommand(ud, d, "update")
+            logger.info("Update "+ ud.url)
+            # update sources there
+            os.chdir(ud.moddir)
+            logger.debug(1, "Running %s", oscupdatecmd)
+            bb.fetch2.check_network_access(d, oscupdatecmd, ud.url)
+            runfetchcmd(oscupdatecmd, d)
+        else:
+            oscfetchcmd = self._buildosccommand(ud, d, "fetch")
+            logger.info("Fetch " + ud.url)
+            # check out sources there
+            bb.utils.mkdirhier(ud.pkgdir)
+            os.chdir(ud.pkgdir)
+            logger.debug(1, "Running %s", oscfetchcmd)
+            bb.fetch2.check_network_access(d, oscfetchcmd, ud.url)
+            runfetchcmd(oscfetchcmd, d)
+
+        os.chdir(os.path.join(ud.pkgdir + ud.path))
+        # tar them up to a defined filename
+        runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d, cleanup = [ud.localpath])
+
+    def supports_srcrev(self):
+        return False
+
+    def generate_config(self, ud, d):
+        """
+        Generate a .oscrc to be used for this run.
+        """
+
+        config_path = os.path.join(d.getVar('OSCDIR', True), "oscrc")
+        if (os.path.exists(config_path)):
+            os.remove(config_path)
+
+        f = open(config_path, 'w')
+        f.write("[general]\n")
+        f.write("apisrv = %s\n" % ud.host)
+        f.write("scheme = http\n")
+        f.write("su-wrapper = su -c\n")
+        f.write("build-root = %s\n" % d.getVar('WORKDIR', True))
+        f.write("urllist = %s\n" % d.getVar("OSCURLLIST", True))
+        f.write("extra-pkgs = gzip\n")
+        f.write("\n")
+        f.write("[%s]\n" % ud.host)
+        f.write("user = %s\n" % ud.parm["user"])
+        f.write("pass = %s\n" % ud.parm["pswd"])
+        f.close()
+
+        return config_path

bitbake/lib/bb/fetch2/perforce.py

@@ -0,0 +1,187 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Fetch' implementations
+
+Classes for obtaining upstream sources for the
+BitBake build tools.
+
+"""
+
+# Copyright (C) 2003, 2004  Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+from future_builtins import zip
+import os
+import subprocess
+import logging
+import bb
+from   bb import data
+from   bb.fetch2 import FetchMethod
+from   bb.fetch2 import FetchError
+from   bb.fetch2 import logger
+from   bb.fetch2 import runfetchcmd
+
+class Perforce(FetchMethod):
+    def supports(self, ud, d):
+        return ud.type in ['p4']
+
+    def doparse(url, d):
+        parm = {}
+        path = url.split("://")[1]
+        delim = path.find("@");
+        if delim != -1:
+            (user, pswd, host, port) = path.split('@')[0].split(":")
+            path = path.split('@')[1]
+        else:
+            (host, port) = d.getVar('P4PORT', False).split(':')
+            user = ""
+            pswd = ""
+
+        if path.find(";") != -1:
+            keys=[]
+            values=[]
+            plist = path.split(';')
+            for item in plist:
+                if item.count('='):
+                    (key, value) = item.split('=')
+                    keys.append(key)
+                    values.append(value)
+
+            parm = dict(zip(keys, values))
+        path = "//" + path.split(';')[0]
+        host += ":%s" % (port)
+        parm["cset"] = Perforce.getcset(d, path, host, user, pswd, parm)
+
+        return host, path, user, pswd, parm
+    doparse = staticmethod(doparse)
+
+    def getcset(d, depot, host, user, pswd, parm):
+        p4opt = ""
+        if "cset" in parm:
+            return parm["cset"];
+        if user:
+            p4opt += " -u %s" % (user)
+        if pswd:
+            p4opt += " -P %s" % (pswd)
+        if host:
+            p4opt += " -p %s" % (host)
+
+        p4date = d.getVar("P4DATE", True)
+        if "revision" in parm:
+            depot += "#%s" % (parm["revision"])
+        elif "label" in parm:
+            depot += "@%s" % (parm["label"])
+        elif p4date:
+            depot += "@%s" % (p4date)
+
+        p4cmd = d.getVar('FETCHCMD_p4', True) or "p4"
+        logger.debug(1, "Running %s%s changes -m 1 %s", p4cmd, p4opt, depot)
+        p4file, errors = bb.process.run("%s%s changes -m 1 %s" % (p4cmd, p4opt, depot))
+        cset = p4file.strip()
+        logger.debug(1, "READ %s", cset)
+        if not cset:
+            return -1
+
+        return cset.split(' ')[1]
+    getcset = staticmethod(getcset)
+
+    def urldata_init(self, ud, d):
+        (host, path, user, pswd, parm) = Perforce.doparse(ud.url, d)
+
+        base_path = path.replace('/...', '')
+        base_path = self._strip_leading_slashes(base_path)
+        
+        if "label" in parm:
+            version = parm["label"]
+        else:
+            version = Perforce.getcset(d, path, host, user, pswd, parm)
+
+        ud.localfile = data.expand('%s+%s+%s.tar.gz' % (host, base_path.replace('/', '.'), version), d)
+
+    def download(self, ud, d):
+        """
+        Fetch urls
+        """
+
+        (host, depot, user, pswd, parm) = Perforce.doparse(ud.url, d)
+
+        if depot.find('/...') != -1:
+            path = depot[:depot.find('/...')]
+        else:
+            path = depot[:depot.rfind('/')]
+
+        module = parm.get('module', os.path.basename(path))
+
+        # Get the p4 command
+        p4opt = ""
+        if user:
+            p4opt += " -u %s" % (user)
+
+        if pswd:
+            p4opt += " -P %s" % (pswd)
+
+        if host:
+            p4opt += " -p %s" % (host)
+
+        p4cmd = d.getVar('FETCHCMD_p4', True) or "p4"
+
+        # create temp directory
+        logger.debug(2, "Fetch: creating temporary directory")
+        bb.utils.mkdirhier(d.expand('${WORKDIR}'))
+        mktemp = d.getVar("FETCHCMD_p4mktemp", True) or d.expand("mktemp -d -q '${WORKDIR}/oep4.XXXXXX'")
+        tmpfile, errors = bb.process.run(mktemp)
+        tmpfile = tmpfile.strip()
+        if not tmpfile:
+            raise FetchError("Fetch: unable to create temporary directory.. make sure 'mktemp' is in the PATH.", ud.url)
+
+        if "label" in parm:
+            depot = "%s@%s" % (depot, parm["label"])
+        else:
+            cset = Perforce.getcset(d, depot, host, user, pswd, parm)
+            depot = "%s@%s" % (depot, cset)
+
+        os.chdir(tmpfile)
+        logger.info("Fetch " + ud.url)
+        logger.info("%s%s files %s", p4cmd, p4opt, depot)
+        p4file, errors = bb.process.run("%s%s files %s" % (p4cmd, p4opt, depot))
+        p4file = [f.rstrip() for f in p4file.splitlines()]
+
+        if not p4file:
+            raise FetchError("Fetch: unable to get the P4 files from %s" % depot, ud.url)
+
+        count = 0
+
+        for file in p4file:
+            list = file.split()
+
+            if list[2] == "delete":
+                continue
+
+            dest = list[0][len(path)+1:]
+            where = dest.find("#")
+
+            subprocess.call("%s%s print -o %s/%s %s" % (p4cmd, p4opt, module, dest[:where], list[0]), shell=True)
+            count = count + 1
+
+        if count == 0:
+            logger.error()
+            raise FetchError("Fetch: No files gathered from the P4 fetch", ud.url)
+
+        runfetchcmd("tar -czf %s %s" % (ud.localpath, module), d, cleanup = [ud.localpath])
+        # cleanup
+        bb.utils.prunedir(tmpfile)

bitbake/lib/bb/fetch2/repo.py

@@ -0,0 +1,98 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake "Fetch" repo (git) implementation
+
+"""
+
+# Copyright (C) 2009 Tom Rini <trini@embeddedalley.com>
+#
+# Based on git.py which is:
+#Copyright (C) 2005 Richard Purdie
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import bb
+from   bb    import data
+from   bb.fetch2 import FetchMethod
+from   bb.fetch2 import runfetchcmd
+
+class Repo(FetchMethod):
+    """Class to fetch a module or modules from repo (git) repositories"""
+    def supports(self, ud, d):
+        """
+        Check to see if a given url can be fetched with repo.
+        """
+        return ud.type in ["repo"]
+
+    def urldata_init(self, ud, d):
+        """
+        We don"t care about the git rev of the manifests repository, but
+        we do care about the manifest to use.  The default is "default".
+        We also care about the branch or tag to be used.  The default is
+        "master".
+        """
+
+        ud.proto = ud.parm.get('protocol', 'git')
+        ud.branch = ud.parm.get('branch', 'master')
+        ud.manifest = ud.parm.get('manifest', 'default.xml')
+        if not ud.manifest.endswith('.xml'):
+            ud.manifest += '.xml'
+
+        ud.localfile = data.expand("repo_%s%s_%s_%s.tar.gz" % (ud.host, ud.path.replace("/", "."), ud.manifest, ud.branch), d)
+
+    def download(self, ud, d):
+        """Fetch url"""
+
+        if os.access(os.path.join(data.getVar("DL_DIR", d, True), ud.localfile), os.R_OK):
+            logger.debug(1, "%s already exists (or was stashed). Skipping repo init / sync.", ud.localpath)
+            return
+
+        gitsrcname = "%s%s" % (ud.host, ud.path.replace("/", "."))
+        repodir = data.getVar("REPODIR", d, True) or os.path.join(data.getVar("DL_DIR", d, True), "repo")
+        codir = os.path.join(repodir, gitsrcname, ud.manifest)
+
+        if ud.user:
+            username = ud.user + "@"
+        else:
+            username = ""
+
+        bb.utils.mkdirhier(os.path.join(codir, "repo"))
+        os.chdir(os.path.join(codir, "repo"))
+        if not os.path.exists(os.path.join(codir, "repo", ".repo")):
+            bb.fetch2.check_network_access(d, "repo init -m %s -b %s -u %s://%s%s%s" % (ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), ud.url)
+            runfetchcmd("repo init -m %s -b %s -u %s://%s%s%s" % (ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d)
+
+        bb.fetch2.check_network_access(d, "repo sync %s" % ud.url, ud.url)
+        runfetchcmd("repo sync", d)
+        os.chdir(codir)
+
+        scmdata = ud.parm.get("scmdata", "")
+        if scmdata == "keep":
+            tar_flags = ""
+        else:
+            tar_flags = "--exclude '.repo' --exclude '.git'"
+
+        # Create a cache
+        runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.join(".", "*") ), d)
+
+    def supports_srcrev(self):
+        return False
+
+    def _build_revision(self, ud, d):
+        return ud.manifest
+
+    def _want_sortable_revision(self, ud, d):
+        return False

bitbake/lib/bb/fetch2/sftp.py

@@ -0,0 +1,129 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake SFTP Fetch implementation
+
+Class for fetching files via SFTP. It tries to adhere to the (now
+expired) IETF Internet Draft for "Uniform Resource Identifier (URI)
+Scheme for Secure File Transfer Protocol (SFTP) and Secure Shell
+(SSH)" (SECSH URI).
+
+It uses SFTP (as to adhere to the SECSH URI specification). It only
+supports key based authentication, not password. This class, unlike
+the SSH fetcher, does not support fetching a directory tree from the
+remote.
+
+  http://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri-04
+  https://www.iana.org/assignments/uri-schemes/prov/sftp
+  https://tools.ietf.org/html/draft-ietf-secsh-filexfer-13
+
+Please note that '/' is used as host path seperator, and not ":"
+as you may be used to from the scp/sftp commands. You can use a
+~ (tilde) to specify a path relative to your home directory.
+(The /~user/ syntax, for specyfing a path relative to another
+user's home directory is not supported.) Note that the tilde must
+still follow the host path seperator ("/"). See exampels below.
+
+Example SRC_URIs:
+
+SRC_URI = "sftp://host.example.com/dir/path.file.txt"
+
+A path relative to your home directory.
+
+SRC_URI = "sftp://host.example.com/~/dir/path.file.txt"
+
+You can also specify a username (specyfing password in the
+URI is not supported, use SSH keys to authenticate):
+
+SRC_URI = "sftp://user@host.example.com/dir/path.file.txt"
+
+"""
+
+# Copyright (C) 2013, Olof Johansson <olof.johansson@axis.com>
+#
+# Based in part on bb.fetch2.wget:
+#    Copyright (C) 2003, 2004  Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+import os
+import bb
+import urllib
+import commands
+from bb import data
+from bb.fetch2 import URI
+from bb.fetch2 import FetchMethod
+from bb.fetch2 import runfetchcmd
+
+
+class SFTP(FetchMethod):
+    """Class to fetch urls via 'sftp'"""
+
+    def supports(self, ud, d):
+        """
+        Check to see if a given url can be fetched with sftp.
+        """
+        return ud.type in ['sftp']
+
+    def recommends_checksum(self, urldata):
+        return True
+
+    def urldata_init(self, ud, d):
+        if 'protocol' in ud.parm and ud.parm['protocol'] == 'git':
+            raise bb.fetch2.ParameterError(
+                "Invalid protocol - if you wish to fetch from a " +
+                "git repository using ssh, you need to use the " +
+                "git:// prefix with protocol=ssh", ud.url)
+
+        if 'downloadfilename' in ud.parm:
+            ud.basename = ud.parm['downloadfilename']
+        else:
+            ud.basename = os.path.basename(ud.path)
+
+        ud.localfile = data.expand(urllib.unquote(ud.basename), d)
+
+    def download(self, ud, d):
+        """Fetch urls"""
+
+        urlo = URI(ud.url)
+        basecmd = 'sftp -oBatchMode=yes'
+        port = ''
+        if urlo.port:
+            port = '-P %d' % urlo.port
+            urlo.port = None
+
+        dldir = data.getVar('DL_DIR', d, True)
+        lpath = os.path.join(dldir, ud.localfile)
+
+        user = ''
+        if urlo.userinfo:
+            user = urlo.userinfo + '@'
+
+        path = urlo.path
+
+        # Supoprt URIs relative to the user's home directory, with
+        # the tilde syntax. (E.g. <sftp://example.com/~/foo.diff>).
+        if path[:3] == '/~/':
+            path = path[3:]
+
+        remote = '%s%s:%s' % (user, urlo.hostname, path)
+
+        cmd = '%s %s %s %s' % (basecmd, port, commands.mkarg(remote),
+                               commands.mkarg(lpath))
+
+        bb.fetch2.check_network_access(d, cmd, ud.url)
+        runfetchcmd(cmd, d)
+        return True

bitbake/lib/bb/fetch2/ssh.py

@@ -0,0 +1,128 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+'''
+BitBake 'Fetch' implementations
+
+This implementation is for Secure Shell (SSH), and attempts to comply with the
+IETF secsh internet draft:
+    http://tools.ietf.org/wg/secsh/draft-ietf-secsh-scp-sftp-ssh-uri/
+
+    Currently does not support the sftp parameters, as this uses scp
+    Also does not support the 'fingerprint' connection parameter.
+
+    Please note that '/' is used as host, path separator not ':' as you may 
+    be used to, also '~' can be used to specify user HOME, but again after '/'
+    
+    Example SRC_URI:
+    SRC_URI = "ssh://user@host.example.com/dir/path/file.txt"
+    SRC_URI = "ssh://user@host.example.com/~/file.txt"
+'''
+
+# Copyright (C) 2006  OpenedHand Ltd.
+#
+#
+# Based in part on svk.py:
+#    Copyright (C) 2006 Holger Hans Peter Freyther
+#    Based on svn.py:
+#        Copyright (C) 2003, 2004  Chris Larson
+#        Based on functions from the base bb module:
+#            Copyright 2003 Holger Schurig
+#
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import re, os
+from   bb import data
+from   bb.fetch2 import FetchMethod
+from   bb.fetch2 import FetchError
+from   bb.fetch2 import logger
+from   bb.fetch2 import runfetchcmd
+
+
+__pattern__ = re.compile(r'''
+ \s*                 # Skip leading whitespace
+ ssh://              # scheme
+ (                   # Optional username/password block
+  (?P<user>\S+)      # username
+  (:(?P<pass>\S+))?  # colon followed by the password (optional)
+ )?
+ (?P<cparam>(;[^;]+)*)?  # connection parameters block (optional)
+ @
+ (?P<host>\S+?)          # non-greedy match of the host
+ (:(?P<port>[0-9]+))?    # colon followed by the port (optional)
+ /
+ (?P<path>[^;]+)         # path on the remote system, may be absolute or relative,
+                         # and may include the use of '~' to reference the remote home
+                         # directory
+ (?P<sparam>(;[^;]+)*)?  # parameters block (optional)
+ $
+''', re.VERBOSE)
+
+class SSH(FetchMethod):
+    '''Class to fetch a module or modules via Secure Shell'''
+
+    def supports(self, urldata, d):
+        return __pattern__.match(urldata.url) != None
+
+    def supports_checksum(self, urldata):
+        return False
+
+    def urldata_init(self, urldata, d):
+        if 'protocol' in urldata.parm and urldata.parm['protocol'] == 'git':
+            raise bb.fetch2.ParameterError(
+                "Invalid protocol - if you wish to fetch from a git " +
+                "repository using ssh, you need to use " +
+                "git:// prefix with protocol=ssh", urldata.url)
+        m = __pattern__.match(urldata.url)
+        path = m.group('path')
+        host = m.group('host')
+        urldata.localpath = os.path.join(d.getVar('DL_DIR', True),
+                os.path.basename(os.path.normpath(path)))
+
+    def download(self, urldata, d):
+        dldir = d.getVar('DL_DIR', True)
+
+        m = __pattern__.match(urldata.url)
+        path = m.group('path')
+        host = m.group('host')
+        port = m.group('port')
+        user = m.group('user')
+        password = m.group('pass')
+
+        if port:
+            portarg = '-P %s' % port
+        else:
+            portarg = ''
+
+        if user:
+            fr = user
+            if password:
+                fr += ':%s' % password
+            fr += '@%s' % host
+        else:
+            fr = host
+        fr += ':%s' % path
+
+
+        import commands
+        cmd = 'scp -B -r %s %s %s/' % (
+            portarg,
+            commands.mkarg(fr),
+            commands.mkarg(dldir)
+        )
+
+        bb.fetch2.check_network_access(d, cmd, urldata.url)
+
+        runfetchcmd(cmd, d)
+

bitbake/lib/bb/fetch2/svn.py

@@ -0,0 +1,197 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Fetch' implementation for svn.
+
+"""
+
+# Copyright (C) 2003, 2004  Chris Larson
+# Copyright (C) 2004        Marcin Juszkiewicz
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+import os
+import sys
+import logging
+import bb
+import re
+from   bb import data
+from   bb.fetch2 import FetchMethod
+from   bb.fetch2 import FetchError
+from   bb.fetch2 import MissingParameterError
+from   bb.fetch2 import runfetchcmd
+from   bb.fetch2 import logger
+
+class Svn(FetchMethod):
+    """Class to fetch a module or modules from svn repositories"""
+    def supports(self, ud, d):
+        """
+        Check to see if a given url can be fetched with svn.
+        """
+        return ud.type in ['svn']
+
+    def urldata_init(self, ud, d):
+        """
+        init svn specific variable within url data
+        """
+        if not "module" in ud.parm:
+            raise MissingParameterError('module', ud.url)
+
+        ud.basecmd = d.getVar('FETCHCMD_svn', True)
+
+        ud.module = ud.parm["module"]
+
+        if not "path_spec" in ud.parm:
+            ud.path_spec = ud.module
+        else:
+            ud.path_spec = ud.parm["path_spec"]
+
+        # Create paths to svn checkouts
+        relpath = self._strip_leading_slashes(ud.path)
+        ud.pkgdir = os.path.join(data.expand('${SVNDIR}', d), ud.host, relpath)
+        ud.moddir = os.path.join(ud.pkgdir, ud.module)
+
+        ud.setup_revisons(d)
+
+        if 'rev' in ud.parm:
+            ud.revision = ud.parm['rev']
+
+        ud.localfile = data.expand('%s_%s_%s_%s_.tar.gz' % (ud.module.replace('/', '.'), ud.host, ud.path.replace('/', '.'), ud.revision), d)
+
+    def _buildsvncommand(self, ud, d, command):
+        """
+        Build up an svn commandline based on ud
+        command is "fetch", "update", "info"
+        """
+
+        proto = ud.parm.get('protocol', 'svn')
+
+        svn_rsh = None
+        if proto == "svn+ssh" and "rsh" in ud.parm:
+            svn_rsh = ud.parm["rsh"]
+
+        svnroot = ud.host + ud.path
+
+        options = []
+
+        options.append("--no-auth-cache")
+
+        if ud.user:
+            options.append("--username %s" % ud.user)
+
+        if ud.pswd:
+            options.append("--password %s" % ud.pswd)
+
+        if command == "info":
+            svncmd = "%s info %s %s://%s/%s/" % (ud.basecmd, " ".join(options), proto, svnroot, ud.module)
+        elif command == "log1":
+            svncmd = "%s log --limit 1 %s %s://%s/%s/" % (ud.basecmd, " ".join(options), proto, svnroot, ud.module)
+        else:
+            suffix = ""
+            if ud.revision:
+                options.append("-r %s" % ud.revision)
+                suffix = "@%s" % (ud.revision)
+
+            if command == "fetch":
+                transportuser = ud.parm.get("transportuser", "")
+                svncmd = "%s co %s %s://%s%s/%s%s %s" % (ud.basecmd, " ".join(options), proto, transportuser, svnroot, ud.module, suffix, ud.path_spec)
+            elif command == "update":
+                svncmd = "%s update %s" % (ud.basecmd, " ".join(options))
+            else:
+                raise FetchError("Invalid svn command %s" % command, ud.url)
+
+        if svn_rsh:
+            svncmd = "svn_RSH=\"%s\" %s" % (svn_rsh, svncmd)
+
+        return svncmd
+
+    def download(self, ud, d):
+        """Fetch url"""
+
+        logger.debug(2, "Fetch: checking for module directory '" + ud.moddir + "'")
+
+        if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK):
+            svnupdatecmd = self._buildsvncommand(ud, d, "update")
+            logger.info("Update " + ud.url)
+            # update sources there
+            os.chdir(ud.moddir)
+            # We need to attempt to run svn upgrade first in case its an older working format
+            try:
+                runfetchcmd(ud.basecmd + " upgrade", d)
+            except FetchError:
+                pass
+            logger.debug(1, "Running %s", svnupdatecmd)
+            bb.fetch2.check_network_access(d, svnupdatecmd, ud.url)
+            runfetchcmd(svnupdatecmd, d)
+        else:
+            svnfetchcmd = self._buildsvncommand(ud, d, "fetch")
+            logger.info("Fetch " + ud.url)
+            # check out sources there
+            bb.utils.mkdirhier(ud.pkgdir)
+            os.chdir(ud.pkgdir)
+            logger.debug(1, "Running %s", svnfetchcmd)
+            bb.fetch2.check_network_access(d, svnfetchcmd, ud.url)
+            runfetchcmd(svnfetchcmd, d)
+
+        scmdata = ud.parm.get("scmdata", "")
+        if scmdata == "keep":
+            tar_flags = ""
+        else:
+            tar_flags = "--exclude '.svn'"
+
+        os.chdir(ud.pkgdir)
+        # tar them up to a defined filename
+        runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.path_spec), d, cleanup = [ud.localpath])
+
+    def clean(self, ud, d):
+        """ Clean SVN specific files and dirs """
+
+        bb.utils.remove(ud.localpath)
+        bb.utils.remove(ud.moddir, True)
+        
+
+    def supports_srcrev(self):
+        return True
+
+    def _revision_key(self, ud, d, name):
+        """
+        Return a unique key for the url
+        """
+        return "svn:" + ud.moddir
+
+    def _latest_revision(self, ud, d, name):
+        """
+        Return the latest upstream revision number
+        """
+        bb.fetch2.check_network_access(d, self._buildsvncommand(ud, d, "log1"))
+
+        output = runfetchcmd("LANG=C LC_ALL=C " + self._buildsvncommand(ud, d, "log1"), d, True)
+
+        # skip the first line, as per output of svn log
+        # then we expect the revision on the 2nd line
+        revision = re.search('^r([0-9]*)', output.splitlines()[1]).group(1)
+
+        return revision
+
+    def sortable_revision(self, ud, d, name):
+        """
+        Return a sortable revision number which in our case is the revision number
+        """
+
+        return False, self._build_revision(ud, d)
+
+    def _build_revision(self, ud, d):
+        return ud.revision

bitbake/lib/bb/fetch2/wget.py

@@ -0,0 +1,565 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'Fetch' implementations
+
+Classes for obtaining upstream sources for the
+BitBake build tools.
+
+"""
+
+# Copyright (C) 2003, 2004  Chris Larson
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+
+import re
+import tempfile
+import subprocess
+import os
+import logging
+import bb
+import urllib
+from   bb import data
+from   bb.fetch2 import FetchMethod
+from   bb.fetch2 import FetchError
+from   bb.fetch2 import logger
+from   bb.fetch2 import runfetchcmd
+from   bb.utils import export_proxies
+from   bs4 import BeautifulSoup
+from   bs4 import SoupStrainer
+
+class Wget(FetchMethod):
+    """Class to fetch urls via 'wget'"""
+    def supports(self, ud, d):
+        """
+        Check to see if a given url can be fetched with wget.
+        """
+        return ud.type in ['http', 'https', 'ftp']
+
+    def recommends_checksum(self, urldata):
+        return True
+
+    def urldata_init(self, ud, d):
+        if 'protocol' in ud.parm:
+            if ud.parm['protocol'] == 'git':
+                raise bb.fetch2.ParameterError("Invalid protocol - if you wish to fetch from a git repository using http, you need to instead use the git:// prefix with protocol=http", ud.url)
+
+        if 'downloadfilename' in ud.parm:
+            ud.basename = ud.parm['downloadfilename']
+        else:
+            ud.basename = os.path.basename(ud.path)
+
+        ud.localfile = data.expand(urllib.unquote(ud.basename), d)
+        if not ud.localfile:
+            ud.localfile = data.expand(urllib.unquote(ud.host + ud.path).replace("/", "."), d)
+
+        self.basecmd = d.getVar("FETCHCMD_wget", True) or "/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate"
+
+    def _runwget(self, ud, d, command, quiet):
+
+        logger.debug(2, "Fetching %s using command '%s'" % (ud.url, command))
+        bb.fetch2.check_network_access(d, command)
+        runfetchcmd(command, d, quiet)
+
+    def download(self, ud, d):
+        """Fetch urls"""
+
+        fetchcmd = self.basecmd
+
+        if 'downloadfilename' in ud.parm:
+            dldir = d.getVar("DL_DIR", True)
+            bb.utils.mkdirhier(os.path.dirname(dldir + os.sep + ud.localfile))
+            fetchcmd += " -O " + dldir + os.sep + ud.localfile
+
+        uri = ud.url.split(";")[0]
+        if os.path.exists(ud.localpath):
+            # file exists, but we didnt complete it.. trying again..
+            fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
+        else:
+            fetchcmd += d.expand(" -P ${DL_DIR} '%s'" % uri)
+
+        self._runwget(ud, d, fetchcmd, False)
+
+        # Sanity check since wget can pretend it succeed when it didn't
+        # Also, this used to happen if sourceforge sent us to the mirror page
+        if not os.path.exists(ud.localpath):
+            raise FetchError("The fetch command returned success for url %s but %s doesn't exist?!" % (uri, ud.localpath), uri)
+
+        if os.path.getsize(ud.localpath) == 0:
+            os.remove(ud.localpath)
+            raise FetchError("The fetch of %s resulted in a zero size file?! Deleting and failing since this isn't right." % (uri), uri)
+
+        return True
+
+    def checkstatus(self, fetch, ud, d):
+        import urllib2, socket, httplib
+        from urllib import addinfourl
+        from bb.fetch2 import FetchConnectionCache
+
+        class HTTPConnectionCache(httplib.HTTPConnection):
+            if fetch.connection_cache:
+                def connect(self):
+                    """Connect to the host and port specified in __init__."""
+
+                    sock = fetch.connection_cache.get_connection(self.host, self.port)
+                    if sock:
+                        self.sock = sock
+                    else:
+                        self.sock = socket.create_connection((self.host, self.port),
+                                    self.timeout, self.source_address)
+                        fetch.connection_cache.add_connection(self.host, self.port, self.sock)
+
+                    if self._tunnel_host:
+                        self._tunnel()
+
+        class CacheHTTPHandler(urllib2.HTTPHandler):
+            def http_open(self, req):
+                return self.do_open(HTTPConnectionCache, req)
+
+            def do_open(self, http_class, req):
+                """Return an addinfourl object for the request, using http_class.
+
+                http_class must implement the HTTPConnection API from httplib.
+                The addinfourl return value is a file-like object.  It also
+                has methods and attributes including:
+                    - info(): return a mimetools.Message object for the headers
+                    - geturl(): return the original request URL
+                    - code: HTTP status code
+                """
+                host = req.get_host()
+                if not host:
+                    raise urlllib2.URLError('no host given')
+
+                h = http_class(host, timeout=req.timeout) # will parse host:port
+                h.set_debuglevel(self._debuglevel)
+
+                headers = dict(req.unredirected_hdrs)
+                headers.update(dict((k, v) for k, v in req.headers.items()
+                            if k not in headers))
+
+                # We want to make an HTTP/1.1 request, but the addinfourl
+                # class isn't prepared to deal with a persistent connection.
+                # It will try to read all remaining data from the socket,
+                # which will block while the server waits for the next request.
+                # So make sure the connection gets closed after the (only)
+                # request.
+
+                # Don't close connection when connection_cache is enabled,
+                if fetch.connection_cache is None: 
+                    headers["Connection"] = "close"
+                else:
+                    headers["Connection"] = "Keep-Alive" # Works for HTTP/1.0
+
+                headers = dict(
+                    (name.title(), val) for name, val in headers.items())
+
+                if req._tunnel_host:
+                    tunnel_headers = {}
+                    proxy_auth_hdr = "Proxy-Authorization"
+                    if proxy_auth_hdr in headers:
+                        tunnel_headers[proxy_auth_hdr] = headers[proxy_auth_hdr]
+                        # Proxy-Authorization should not be sent to origin
+                        # server.
+                        del headers[proxy_auth_hdr]
+                    h.set_tunnel(req._tunnel_host, headers=tunnel_headers)
+
+                try:
+                    h.request(req.get_method(), req.get_selector(), req.data, headers)
+                except socket.error, err: # XXX what error?
+                    # Don't close connection when cache is enabled.
+                    if fetch.connection_cache is None:
+                        h.close()
+                    raise urllib2.URLError(err)
+                else:
+                    try:
+                        r = h.getresponse(buffering=True)
+                    except TypeError: # buffering kw not supported
+                        r = h.getresponse()
+
+                # Pick apart the HTTPResponse object to get the addinfourl
+                # object initialized properly.
+
+                # Wrap the HTTPResponse object in socket's file object adapter
+                # for Windows.  That adapter calls recv(), so delegate recv()
+                # to read().  This weird wrapping allows the returned object to
+                # have readline() and readlines() methods.
+
+                # XXX It might be better to extract the read buffering code
+                # out of socket._fileobject() and into a base class.
+                r.recv = r.read
+
+                # no data, just have to read
+                r.read()
+                class fp_dummy(object):
+                    def read(self):
+                        return ""
+                    def readline(self):
+                        return ""
+                    def close(self):
+                        pass
+
+                resp = addinfourl(fp_dummy(), r.msg, req.get_full_url())
+                resp.code = r.status
+                resp.msg = r.reason
+
+                # Close connection when server request it.
+                if fetch.connection_cache is not None:
+                    if 'Connection' in r.msg and r.msg['Connection'] == 'close':
+                        fetch.connection_cache.remove_connection(h.host, h.port)
+
+                return resp
+
+        class HTTPMethodFallback(urllib2.BaseHandler):
+            """
+            Fallback to GET if HEAD is not allowed (405 HTTP error)
+            """
+            def http_error_405(self, req, fp, code, msg, headers):
+                fp.read()
+                fp.close()
+
+                newheaders = dict((k,v) for k,v in req.headers.items()
+                                  if k.lower() not in ("content-length", "content-type"))
+                return self.parent.open(urllib2.Request(req.get_full_url(),
+                                                        headers=newheaders,
+                                                        origin_req_host=req.get_origin_req_host(),
+                                                        unverifiable=True))
+
+            """
+            Some servers (e.g. GitHub archives, hosted on Amazon S3) return 403
+            Forbidden when they actually mean 405 Method Not Allowed.
+            """
+            http_error_403 = http_error_405
+
+            """
+            Some servers (e.g. FusionForge) returns 406 Not Acceptable when they
+            actually mean 405 Method Not Allowed.
+            """
+            http_error_406 = http_error_405
+
+        class FixedHTTPRedirectHandler(urllib2.HTTPRedirectHandler):
+            """
+            urllib2.HTTPRedirectHandler resets the method to GET on redirect,
+            when we want to follow redirects using the original method.
+            """
+            def redirect_request(self, req, fp, code, msg, headers, newurl):
+                newreq = urllib2.HTTPRedirectHandler.redirect_request(self, req, fp, code, msg, headers, newurl)
+                newreq.get_method = lambda: req.get_method()
+                return newreq
+        exported_proxies = export_proxies(d)
+
+        handlers = [FixedHTTPRedirectHandler, HTTPMethodFallback]
+        if export_proxies:
+            handlers.append(urllib2.ProxyHandler())
+        handlers.append(CacheHTTPHandler())
+        # XXX: Since Python 2.7.9 ssl cert validation is enabled by default
+        # see PEP-0476, this causes verification errors on some https servers
+        # so disable by default.
+        import ssl
+        if hasattr(ssl, '_create_unverified_context'):
+            handlers.append(urllib2.HTTPSHandler(context=ssl._create_unverified_context()))
+        opener = urllib2.build_opener(*handlers)
+
+        try:
+            uri = ud.url.split(";")[0]
+            r = urllib2.Request(uri)
+            r.get_method = lambda: "HEAD"
+            opener.open(r)
+        except urllib2.URLError as e:
+            # debug for now to avoid spamming the logs in e.g. remote sstate searches
+            logger.debug(2, "checkstatus() urlopen failed: %s" % e)
+            return False
+        return True
+
+    def _parse_path(self, regex, s):
+        """
+        Find and group name, version and archive type in the given string s
+        """
+
+        m = regex.search(s)
+        if m:
+            pname = ''
+            pver = ''
+            ptype = ''
+
+            mdict = m.groupdict()
+            if 'name' in mdict.keys():
+                pname = mdict['name']
+            if 'pver' in mdict.keys():
+                pver = mdict['pver']
+            if 'type' in mdict.keys():
+                ptype = mdict['type']
+
+            bb.debug(3, "_parse_path: %s, %s, %s" % (pname, pver, ptype))
+
+            return (pname, pver, ptype)
+
+        return None
+
+    def _modelate_version(self, version):
+        if version[0] in ['.', '-']:
+            if version[1].isdigit():
+                version = version[1] + version[0] + version[2:len(version)]
+            else:
+                version = version[1:len(version)]
+
+        version = re.sub('-', '.', version)
+        version = re.sub('_', '.', version)
+        version = re.sub('(rc)+', '.1000.', version)
+        version = re.sub('(beta)+', '.100.', version)
+        version = re.sub('(alpha)+', '.10.', version)
+        if version[0] == 'v':
+            version = version[1:len(version)]
+        return version
+
+    def _vercmp(self, old, new):
+        """
+        Check whether 'new' is newer than 'old' version. We use existing vercmp() for the
+        purpose. PE is cleared in comparison as it's not for build, and PR is cleared too
+        for simplicity as it's somehow difficult to get from various upstream format
+        """
+
+        (oldpn, oldpv, oldsuffix) = old
+        (newpn, newpv, newsuffix) = new
+
+        """
+        Check for a new suffix type that we have never heard of before
+        """
+        if (newsuffix):
+            m = self.suffix_regex_comp.search(newsuffix)
+            if not m:
+                bb.warn("%s has a possible unknown suffix: %s" % (newpn, newsuffix))
+                return False
+
+        """
+        Not our package so ignore it
+        """
+        if oldpn != newpn:
+            return False
+
+        oldpv = self._modelate_version(oldpv)
+        newpv = self._modelate_version(newpv)
+
+        return bb.utils.vercmp(("0", oldpv, ""), ("0", newpv, ""))
+
+    def _fetch_index(self, uri, ud, d):
+        """
+        Run fetch checkstatus to get directory information
+        """
+        f = tempfile.NamedTemporaryFile()
+
+        agent = "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Ubuntu/9.10 (karmic) Firefox/3.6.12"
+        fetchcmd = self.basecmd
+        fetchcmd += " -O " + f.name + " --user-agent='" + agent + "' '" + uri + "'"
+        try:
+            self._runwget(ud, d, fetchcmd, True)
+            fetchresult = f.read()
+        except bb.fetch2.BBFetchException:
+            fetchresult = ""
+
+        f.close()
+        return fetchresult
+
+    def _check_latest_version(self, url, package, package_regex, current_version, ud, d):
+        """
+        Return the latest version of a package inside a given directory path
+        If error or no version, return ""
+        """
+        valid = 0
+        version = ['', '', '']
+
+        bb.debug(3, "VersionURL: %s" % (url))
+        soup = BeautifulSoup(self._fetch_index(url, ud, d), "html.parser", parse_only=SoupStrainer("a"))
+        if not soup:
+            bb.debug(3, "*** %s NO SOUP" % (url))
+            return ""
+
+        for line in soup.find_all('a', href=True):
+            bb.debug(3, "line['href'] = '%s'" % (line['href']))
+            bb.debug(3, "line = '%s'" % (str(line)))
+
+            newver = self._parse_path(package_regex, line['href'])
+            if not newver:
+                newver = self._parse_path(package_regex, str(line))
+
+            if newver:
+                bb.debug(3, "Upstream version found: %s" % newver[1])
+                if valid == 0:
+                    version = newver
+                    valid = 1
+                elif self._vercmp(version, newver) < 0:
+                    version = newver
+                
+        pupver = re.sub('_', '.', version[1])
+
+        bb.debug(3, "*** %s -> UpstreamVersion = %s (CurrentVersion = %s)" %
+                (package, pupver or "N/A", current_version[1]))
+
+        if valid:
+            return pupver
+
+        return ""
+
+    def _check_latest_version_by_dir(self, dirver, package, package_regex,
+            current_version, ud, d):
+        """
+            Scan every directory in order to get upstream version.
+        """
+        version_dir = ['', '', '']
+        version = ['', '', '']
+
+        dirver_regex = re.compile("(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])+(\d+))")
+        s = dirver_regex.search(dirver)
+        if s:
+            version_dir[1] = s.group('ver')
+        else:
+            version_dir[1] = dirver
+
+        dirs_uri = bb.fetch.encodeurl([ud.type, ud.host,
+                ud.path.split(dirver)[0], ud.user, ud.pswd, {}])
+        bb.debug(3, "DirURL: %s, %s" % (dirs_uri, package))
+
+        soup = BeautifulSoup(self._fetch_index(dirs_uri, ud, d), "html.parser", parse_only=SoupStrainer("a"))
+        if not soup:
+            return version[1]
+
+        for line in soup.find_all('a', href=True):
+            s = dirver_regex.search(line['href'].strip("/"))
+            if s:
+                sver = s.group('ver')
+
+                # When prefix is part of the version directory it need to
+                # ensure that only version directory is used so remove previous
+                # directories if exists.
+                #
+                # Example: pfx = '/dir1/dir2/v' and version = '2.5' the expected
+                # result is v2.5.
+                spfx = s.group('pfx').split('/')[-1]
+
+                version_dir_new = ['', sver, '']
+                if self._vercmp(version_dir, version_dir_new) <= 0:
+                    dirver_new = spfx + sver
+                    path = ud.path.replace(dirver, dirver_new, True) \
+                        .split(package)[0]
+                    uri = bb.fetch.encodeurl([ud.type, ud.host, path,
+                        ud.user, ud.pswd, {}])
+
+                    pupver = self._check_latest_version(uri,
+                            package, package_regex, current_version, ud, d)
+                    if pupver:
+                        version[1] = pupver
+
+                    version_dir = version_dir_new
+
+        return version[1]
+
+    def _init_regexes(self, package, ud, d):
+        """
+        Match as many patterns as possible such as:
+                gnome-common-2.20.0.tar.gz (most common format)
+                gtk+-2.90.1.tar.gz
+                xf86-input-synaptics-12.6.9.tar.gz
+                dri2proto-2.3.tar.gz
+                blktool_4.orig.tar.gz
+                libid3tag-0.15.1b.tar.gz
+                unzip552.tar.gz
+                icu4c-3_6-src.tgz
+                genext2fs_1.3.orig.tar.gz
+                gst-fluendo-mp3
+        """
+        # match most patterns which uses "-" as separator to version digits
+        pn_prefix1 = "[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]"
+        # a loose pattern such as for unzip552.tar.gz
+        pn_prefix2 = "[a-zA-Z]+"
+        # a loose pattern such as for 80325-quicky-0.4.tar.gz
+        pn_prefix3 = "[0-9]+[-]?[a-zA-Z]+"
+        # Save the Package Name (pn) Regex for use later
+        pn_regex = "(%s|%s|%s)" % (pn_prefix1, pn_prefix2, pn_prefix3)
+
+        # match version
+        pver_regex = "(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)"
+
+        # match arch
+        parch_regex = "-source|_all_"
+
+        # src.rpm extension was added only for rpm package. Can be removed if the rpm
+        # packaged will always be considered as having to be manually upgraded
+        psuffix_regex = "(tar\.gz|tgz|tar\.bz2|zip|xz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
+
+        # match name, version and archive type of a package
+        package_regex_comp = re.compile("(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
+                                                    % (pn_regex, pver_regex, parch_regex, psuffix_regex))
+        self.suffix_regex_comp = re.compile(psuffix_regex)
+
+        # compile regex, can be specific by package or generic regex
+        pn_regex = d.getVar('UPSTREAM_CHECK_REGEX', True)
+        if pn_regex:
+            package_custom_regex_comp = re.compile(pn_regex)
+        else:
+            version = self._parse_path(package_regex_comp, package)
+            if version:
+                package_custom_regex_comp = re.compile(
+                    "(?P<name>%s)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s)" %
+                    (re.escape(version[0]), pver_regex, parch_regex, psuffix_regex))
+            else:
+                package_custom_regex_comp = None
+
+        return package_custom_regex_comp
+
+    def latest_versionstring(self, ud, d):
+        """
+        Manipulate the URL and try to obtain the latest package version
+
+        sanity check to ensure same name and type.
+        """
+        package = ud.path.split("/")[-1]
+        current_version = ['', d.getVar('PV', True), '']
+
+        """possible to have no version in pkg name, such as spectrum-fw"""
+        if not re.search("\d+", package):
+            current_version[1] = re.sub('_', '.', current_version[1])
+            current_version[1] = re.sub('-', '.', current_version[1])
+            return (current_version[1], '')
+
+        package_regex = self._init_regexes(package, ud, d)
+        if package_regex is None:
+            bb.warn("latest_versionstring: package %s don't match pattern" % (package))
+            return ('', '')
+        bb.debug(3, "latest_versionstring, regex: %s" % (package_regex.pattern))
+
+        uri = ""
+        regex_uri = d.getVar("UPSTREAM_CHECK_URI", True)
+        if not regex_uri:
+            path = ud.path.split(package)[0]
+
+            # search for version matches on folders inside the path, like:
+            # "5.7" in http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz
+            dirver_regex = re.compile("(?P<dirver>[^/]*(\d+\.)*\d+([-_]r\d+)*)/")
+            m = dirver_regex.search(path)
+            if m:
+                pn = d.getVar('PN', True)
+                dirver = m.group('dirver')
+
+                dirver_pn_regex = re.compile("%s\d?" % (re.escape(pn)))
+                if not dirver_pn_regex.search(dirver):
+                    return (self._check_latest_version_by_dir(dirver,
+                        package, package_regex, current_version, ud, d), '')
+
+            uri = bb.fetch.encodeurl([ud.type, ud.host, path, ud.user, ud.pswd, {}])
+        else:
+            uri = regex_uri
+
+        return (self._check_latest_version(uri, package, package_regex,
+                current_version, ud, d), '')

bitbake/lib/bb/main.py

@@ -0,0 +1,440 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (C) 2003, 2004  Chris Larson
+# Copyright (C) 2003, 2004  Phil Blundell
+# Copyright (C) 2003 - 2005 Michael 'Mickey' Lauer
+# Copyright (C) 2005        Holger Hans Peter Freyther
+# Copyright (C) 2005        ROAD GmbH
+# Copyright (C) 2006        Richard Purdie
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import sys
+import logging
+import optparse
+import warnings
+
+import bb
+from bb import event
+import bb.msg
+from bb import cooker
+from bb import ui
+from bb import server
+from bb import cookerdata
+
+logger = logging.getLogger("BitBake")
+
+class BBMainException(Exception):
+    pass
+
+def present_options(optionlist):
+    if len(optionlist) > 1:
+        return ' or '.join([', '.join(optionlist[:-1]), optionlist[-1]])
+    else:
+        return optionlist[0]
+
+class BitbakeHelpFormatter(optparse.IndentedHelpFormatter):
+    def format_option(self, option):
+        # We need to do this here rather than in the text we supply to
+        # add_option() because we don't want to call list_extension_modules()
+        # on every execution (since it imports all of the modules)
+        # Note also that we modify option.help rather than the returned text
+        # - this is so that we don't have to re-format the text ourselves
+        if option.dest == 'ui':
+            valid_uis = list_extension_modules(bb.ui, 'main')
+            option.help = option.help.replace('@CHOICES@', present_options(valid_uis))
+        elif option.dest == 'servertype':
+            valid_server_types = list_extension_modules(bb.server, 'BitBakeServer')
+            option.help = option.help.replace('@CHOICES@', present_options(valid_server_types))
+
+        return optparse.IndentedHelpFormatter.format_option(self, option)
+
+def list_extension_modules(pkg, checkattr):
+    """
+    Lists extension modules in a specific Python package
+    (e.g. UIs, servers). NOTE: Calling this function will import all of the
+    submodules of the specified module in order to check for the specified
+    attribute; this can have unusual side-effects. As a result, this should
+    only be called when displaying help text or error messages.
+    Parameters:
+        pkg: previously imported Python package to list
+        checkattr: attribute to look for in module to determine if it's valid
+            as the type of extension you are looking for
+    """
+    import pkgutil
+    pkgdir = os.path.dirname(pkg.__file__)
+
+    modules = []
+    for _, modulename, _ in pkgutil.iter_modules([pkgdir]):
+        if os.path.isdir(os.path.join(pkgdir, modulename)):
+            # ignore directories
+            continue
+        try:
+            module = __import__(pkg.__name__, fromlist=[modulename])
+        except:
+            # If we can't import it, it's not valid
+            continue
+        module_if = getattr(module, modulename)
+        if getattr(module_if, 'hidden_extension', False):
+            continue
+        if not checkattr or hasattr(module_if, checkattr):
+            modules.append(modulename)
+    return modules
+
+def import_extension_module(pkg, modulename, checkattr):
+    try:
+        # Dynamically load the UI based on the ui name. Although we
+        # suggest a fixed set this allows you to have flexibility in which
+        # ones are available.
+        module = __import__(pkg.__name__, fromlist = [modulename])
+        return getattr(module, modulename)
+    except AttributeError:
+        raise BBMainException('FATAL: Unable to import extension module "%s" from %s. Valid extension modules: %s' % (modulename, pkg.__name__, present_options(list_extension_modules(pkg, checkattr))))
+
+
+# Display bitbake/OE warnings via the BitBake.Warnings logger, ignoring others"""
+warnlog = logging.getLogger("BitBake.Warnings")
+_warnings_showwarning = warnings.showwarning
+def _showwarning(message, category, filename, lineno, file=None, line=None):
+    if file is not None:
+        if _warnings_showwarning is not None:
+            _warnings_showwarning(message, category, filename, lineno, file, line)
+    else:
+        s = warnings.formatwarning(message, category, filename, lineno)
+        warnlog.warn(s)
+
+warnings.showwarning = _showwarning
+warnings.filterwarnings("ignore")
+warnings.filterwarnings("default", module="(<string>$|(oe|bb)\.)")
+warnings.filterwarnings("ignore", category=PendingDeprecationWarning)
+warnings.filterwarnings("ignore", category=ImportWarning)
+warnings.filterwarnings("ignore", category=DeprecationWarning, module="<string>$")
+warnings.filterwarnings("ignore", message="With-statements now directly support multiple context managers")
+
+class BitBakeConfigParameters(cookerdata.ConfigParameters):
+
+    def parseCommandLine(self, argv=sys.argv):
+        parser = optparse.OptionParser(
+            formatter = BitbakeHelpFormatter(),
+            version = "BitBake Build Tool Core version %s" % bb.__version__,
+            usage = """%prog [options] [recipename/target recipe:do_task ...]
+
+    Executes the specified task (default is 'build') for a given set of target recipes (.bb files).
+    It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which
+    will provide the layer, BBFILES and other configuration information.""")
+
+        parser.add_option("-b", "--buildfile", help = "Execute tasks from a specific .bb recipe directly. WARNING: Does not handle any dependencies from other recipes.",
+                   action = "store", dest = "buildfile", default = None)
+
+        parser.add_option("-k", "--continue", help = "Continue as much as possible after an error. While the target that failed and anything depending on it cannot be built, as much as possible will be built before stopping.",
+                   action = "store_false", dest = "abort", default = True)
+
+        parser.add_option("-a", "--tryaltconfigs", help = "Continue with builds by trying to use alternative providers where possible.",
+                   action = "store_true", dest = "tryaltconfigs", default = False)
+
+        parser.add_option("-f", "--force", help = "Force the specified targets/task to run (invalidating any existing stamp file).",
+                   action = "store_true", dest = "force", default = False)
+
+        parser.add_option("-c", "--cmd", help = "Specify the task to execute. The exact options available depend on the metadata. Some examples might be 'compile' or 'populate_sysroot' or 'listtasks' may give a list of the tasks available.",
+                   action = "store", dest = "cmd")
+
+        parser.add_option("-C", "--clear-stamp", help = "Invalidate the stamp for the specified task such as 'compile' and then run the default task for the specified target(s).",
+                    action = "store", dest = "invalidate_stamp")
+
+        parser.add_option("-r", "--read", help = "Read the specified file before bitbake.conf.",
+                   action = "append", dest = "prefile", default = [])
+
+        parser.add_option("-R", "--postread", help = "Read the specified file after bitbake.conf.",
+                          action = "append", dest = "postfile", default = [])
+
+        parser.add_option("-v", "--verbose", help = "Output more log message data to the terminal.",
+                   action = "store_true", dest = "verbose", default = False)
+
+        parser.add_option("-D", "--debug", help = "Increase the debug level. You can specify this more than once.",
+                   action = "count", dest="debug", default = 0)
+
+        parser.add_option("-n", "--dry-run", help = "Don't execute, just go through the motions.",
+                   action = "store_true", dest = "dry_run", default = False)
+
+        parser.add_option("-S", "--dump-signatures", help = "Dump out the signature construction information, with no task execution. The SIGNATURE_HANDLER parameter is passed to the handler. Two common values are none and printdiff but the handler may define more/less. none means only dump the signature, printdiff means compare the dumped signature with the cached one.",
+                   action = "append", dest = "dump_signatures", default = [], metavar="SIGNATURE_HANDLER")
+
+        parser.add_option("-p", "--parse-only", help = "Quit after parsing the BB recipes.",
+                   action = "store_true", dest = "parse_only", default = False)
+
+        parser.add_option("-s", "--show-versions", help = "Show current and preferred versions of all recipes.",
+                   action = "store_true", dest = "show_versions", default = False)
+
+        parser.add_option("-e", "--environment", help = "Show the global or per-recipe environment complete with information about where variables were set/changed.",
+                   action = "store_true", dest = "show_environment", default = False)
+
+        parser.add_option("-g", "--graphviz", help = "Save dependency tree information for the specified targets in the dot syntax.",
+                    action = "store_true", dest = "dot_graph", default = False)
+
+        parser.add_option("-I", "--ignore-deps", help = """Assume these dependencies don't exist and are already provided (equivalent to ASSUME_PROVIDED). Useful to make dependency graphs more appealing""",
+                    action = "append", dest = "extra_assume_provided", default = [])
+
+        parser.add_option("-l", "--log-domains", help = """Show debug logging for the specified logging domains""",
+                    action = "append", dest = "debug_domains", default = [])
+
+        parser.add_option("-P", "--profile", help = "Profile the command and save reports.",
+                   action = "store_true", dest = "profile", default = False)
+
+        env_ui = os.environ.get('BITBAKE_UI', None)
+        default_ui = env_ui or 'knotty'
+        # @CHOICES@ is substituted out by BitbakeHelpFormatter above
+        parser.add_option("-u", "--ui", help = "The user interface to use (@CHOICES@ - default %default).",
+                   action="store", dest="ui", default=default_ui)
+
+        # @CHOICES@ is substituted out by BitbakeHelpFormatter above
+        parser.add_option("-t", "--servertype", help = "Choose which server type to use (@CHOICES@ - default %default).",
+                   action = "store", dest = "servertype", default = "process")
+
+        parser.add_option("", "--token", help = "Specify the connection token to be used when connecting to a remote server.",
+                   action = "store", dest = "xmlrpctoken")
+
+        parser.add_option("", "--revisions-changed", help = "Set the exit code depending on whether upstream floating revisions have changed or not.",
+                   action = "store_true", dest = "revisions_changed", default = False)
+
+        parser.add_option("", "--server-only", help = "Run bitbake without a UI, only starting a server (cooker) process.",
+                   action = "store_true", dest = "server_only", default = False)
+
+        parser.add_option("-B", "--bind", help = "The name/address for the bitbake server to bind to.",
+                   action = "store", dest = "bind", default = False)
+
+        parser.add_option("", "--no-setscene", help = "Do not run any setscene tasks. sstate will be ignored and everything needed, built.",
+                   action = "store_true", dest = "nosetscene", default = False)
+
+        parser.add_option("", "--setscene-only", help = "Only run setscene tasks, don't run any real tasks.",
+                   action = "store_true", dest = "setsceneonly", default = False)
+
+        parser.add_option("", "--remote-server", help = "Connect to the specified server.",
+                   action = "store", dest = "remote_server", default = False)
+
+        parser.add_option("-m", "--kill-server", help = "Terminate the remote server.",
+                    action = "store_true", dest = "kill_server", default = False)
+
+        parser.add_option("", "--observe-only", help = "Connect to a server as an observing-only client.",
+                   action = "store_true", dest = "observe_only", default = False)
+
+        parser.add_option("", "--status-only", help = "Check the status of the remote bitbake server.",
+                   action = "store_true", dest = "status_only", default = False)
+
+        parser.add_option("-w", "--write-log", help = "Writes the event log of the build to a bitbake event json file. Use '' (empty string) to assign the name automatically.",
+                   action = "store", dest = "writeeventlog")
+
+        options, targets = parser.parse_args(argv)
+
+        # some environmental variables set also configuration options
+        if "BBSERVER" in os.environ:
+            options.servertype = "xmlrpc"
+            options.remote_server = os.environ["BBSERVER"]
+
+        if "BBTOKEN" in os.environ:
+            options.xmlrpctoken = os.environ["BBTOKEN"]
+
+        if "BBEVENTLOG" in os.environ:
+            options.writeeventlog = os.environ["BBEVENTLOG"]
+
+        # fill in proper log name if not supplied
+        if options.writeeventlog is not None and len(options.writeeventlog) == 0:
+            import datetime
+            options.writeeventlog = "bitbake_eventlog_%s.json" % datetime.datetime.now().strftime("%Y%m%d%H%M%S")
+
+        # if BBSERVER says to autodetect, let's do that
+        if options.remote_server:
+            [host, port] = options.remote_server.split(":", 2)
+            port = int(port)
+            # use automatic port if port set to -1, means read it from
+            # the bitbake.lock file; this is a bit tricky, but we always expect
+            # to be in the base of the build directory if we need to have a
+            # chance to start the server later, anyway
+            if port == -1:
+                lock_location = "./bitbake.lock"
+                # we try to read the address at all times; if the server is not started,
+                # we'll try to start it after the first connect fails, below
+                try:
+                    lf = open(lock_location, 'r')
+                    remotedef = lf.readline()
+                    [host, port] = remotedef.split(":")
+                    port = int(port)
+                    lf.close()
+                    options.remote_server = remotedef
+                except Exception as e:
+                    raise BBMainException("Failed to read bitbake.lock (%s), invalid port" % str(e))
+
+        return options, targets[1:]
+
+
+def start_server(servermodule, configParams, configuration, features):
+    server = servermodule.BitBakeServer()
+    single_use = not configParams.server_only
+    if configParams.bind:
+        (host, port) = configParams.bind.split(':')
+        server.initServer((host, int(port)), single_use)
+        configuration.interface = [ server.serverImpl.host, server.serverImpl.port ]
+    else:
+        server.initServer(single_use=single_use)
+        configuration.interface = []
+
+    try:
+        configuration.setServerRegIdleCallback(server.getServerIdleCB())
+
+        cooker = bb.cooker.BBCooker(configuration, features)
+
+        server.addcooker(cooker)
+        server.saveConnectionDetails()
+    except Exception as e:
+        exc_info = sys.exc_info()
+        while hasattr(server, "event_queue"):
+            try:
+                import queue
+            except ImportError:
+                import Queue as queue
+            try:
+                event = server.event_queue.get(block=False)
+            except (queue.Empty, IOError):
+                break
+            if isinstance(event, logging.LogRecord):
+                logger.handle(event)
+        raise exc_info[1], None, exc_info[2]
+    server.detach()
+    cooker.lock.close()
+    return server
+
+
+def bitbake_main(configParams, configuration):
+
+    # Python multiprocessing requires /dev/shm on Linux
+    if sys.platform.startswith('linux') and not os.access('/dev/shm', os.W_OK | os.X_OK):
+        raise BBMainException("FATAL: /dev/shm does not exist or is not writable")
+
+    # Unbuffer stdout to avoid log truncation in the event
+    # of an unorderly exit as well as to provide timely
+    # updates to log files for use with tail
+    try:
+        if sys.stdout.name == '<stdout>':
+            sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
+    except:
+        pass
+
+
+    configuration.setConfigParameters(configParams)
+
+    ui_module = import_extension_module(bb.ui, configParams.ui, 'main')
+    servermodule = import_extension_module(bb.server, configParams.servertype, 'BitBakeServer')
+
+    if configParams.server_only:
+        if configParams.servertype != "xmlrpc":
+            raise BBMainException("FATAL: If '--server-only' is defined, we must set the "
+                                  "servertype as 'xmlrpc'.\n")
+        if not configParams.bind:
+            raise BBMainException("FATAL: The '--server-only' option requires a name/address "
+                                  "to bind to with the -B option.\n")
+        if configParams.remote_server:
+            raise BBMainException("FATAL: The '--server-only' option conflicts with %s.\n" %
+                                  ("the BBSERVER environment variable" if "BBSERVER" in os.environ \
+                                   else "the '--remote-server' option" ))
+
+    if configParams.bind and configParams.servertype != "xmlrpc":
+        raise BBMainException("FATAL: If '-B' or '--bind' is defined, we must "
+                              "set the servertype as 'xmlrpc'.\n")
+
+    if configParams.remote_server and configParams.servertype != "xmlrpc":
+        raise BBMainException("FATAL: If '--remote-server' is defined, we must "
+                              "set the servertype as 'xmlrpc'.\n")
+
+    if configParams.observe_only and (not configParams.remote_server or configParams.bind):
+        raise BBMainException("FATAL: '--observe-only' can only be used by UI clients "
+                              "connecting to a server.\n")
+
+    if configParams.kill_server and not configParams.remote_server:
+        raise BBMainException("FATAL: '--kill-server' can only be used to terminate a remote server")
+
+    if "BBDEBUG" in os.environ:
+        level = int(os.environ["BBDEBUG"])
+        if level > configuration.debug:
+            configuration.debug = level
+
+    bb.msg.init_msgconfig(configParams.verbose, configuration.debug,
+                         configuration.debug_domains)
+
+    # Ensure logging messages get sent to the UI as events
+    handler = bb.event.LogHandler()
+    if not configParams.status_only:
+        # In status only mode there are no logs and no UI
+        logger.addHandler(handler)
+
+    # Clear away any spurious environment variables while we stoke up the cooker
+    cleanedvars = bb.utils.clean_environment()
+
+    featureset = []
+    if not configParams.server_only:
+        # Collect the feature set for the UI
+        featureset = getattr(ui_module, "featureSet", [])
+
+    if configParams.server_only:
+        for param in ('prefile', 'postfile'):
+            value = getattr(configParams, param)
+            if value:
+                setattr(configuration, "%s_server" % param, value)
+                param = "%s_server" % param
+
+    if not configParams.remote_server:
+        # we start a server with a given configuration
+        server = start_server(servermodule, configParams, configuration, featureset)
+        bb.event.ui_queue = []
+    else:
+        # we start a stub server that is actually a XMLRPClient that connects to a real server
+        server = servermodule.BitBakeXMLRPCClient(configParams.observe_only, configParams.xmlrpctoken)
+        server.saveConnectionDetails(configParams.remote_server)
+
+
+    if not configParams.server_only:
+        try:
+            server_connection = server.establishConnection(featureset)
+        except Exception as e:
+            bb.fatal("Could not connect to server %s: %s" % (configParams.remote_server, str(e)))
+
+        if configParams.kill_server:
+            server_connection.connection.terminateServer()
+            bb.event.ui_queue = []
+            return 0
+
+        server_connection.setupEventQueue()
+
+        # Restore the environment in case the UI needs it
+        for k in cleanedvars:
+            os.environ[k] = cleanedvars[k]
+
+        logger.removeHandler(handler)
+
+
+        if configParams.status_only:
+            server_connection.terminate()
+            return 0
+
+        try:
+            return ui_module.main(server_connection.connection, server_connection.events, configParams)
+        finally:
+            bb.event.ui_queue = []
+            server_connection.terminate()
+    else:
+        print("Bitbake server address: %s, server port: %s" % (server.serverImpl.host, server.serverImpl.port))
+        return 0
+
+    return 1

bitbake/lib/bb/methodpool.py

@@ -0,0 +1,40 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+#
+# Copyright (C)       2006 Holger Hans Peter Freyther
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+from bb.utils import better_compile, better_exec
+
+def insert_method(modulename, code, fn, lineno):
+    """
+    Add code of a module should be added. The methods
+    will be simply added, no checking will be done
+    """
+    comp = better_compile(code, modulename, fn, lineno=lineno)
+    better_exec(comp, None, code, fn)
+
+compilecache = {}
+
+def compile_cache(code):
+    h = hash(code)
+    if h in compilecache:
+        return compilecache[h]
+    return None
+
+def compile_cache_add(code, compileobj):
+    h = hash(code)
+    compilecache[h] = compileobj

bitbake/lib/bb/monitordisk.py

@@ -0,0 +1,263 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (C) 2012 Robert Yang
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os, logging, re, sys
+import bb
+logger = logging.getLogger("BitBake.Monitor")
+
+def printErr(info):
+    logger.error("%s\n       Disk space monitor will NOT be enabled" % info)
+
+def convertGMK(unit):
+
+    """ Convert the space unit G, M, K, the unit is case-insensitive """
+
+    unitG = re.match('([1-9][0-9]*)[gG]\s?$', unit)
+    if unitG:
+        return int(unitG.group(1)) * (1024 ** 3)
+    unitM = re.match('([1-9][0-9]*)[mM]\s?$', unit)
+    if unitM:
+        return int(unitM.group(1)) * (1024 ** 2)
+    unitK = re.match('([1-9][0-9]*)[kK]\s?$', unit)
+    if unitK:
+        return int(unitK.group(1)) * 1024
+    unitN = re.match('([1-9][0-9]*)\s?$', unit)
+    if unitN:
+        return int(unitN.group(1))
+    else:
+        return None
+
+def getMountedDev(path):
+
+    """ Get the device mounted at the path, uses /proc/mounts """
+
+    # Get the mount point of the filesystem containing path
+    # st_dev is the ID of device containing file
+    parentDev = os.stat(path).st_dev
+    currentDev = parentDev
+    # When the current directory's device is different from the
+    # parent's, then the current directory is a mount point
+    while parentDev == currentDev:
+        mountPoint = path
+        # Use dirname to get the parent's directory
+        path = os.path.dirname(path)
+        # Reach the "/"
+        if path == mountPoint:
+            break
+        parentDev= os.stat(path).st_dev
+
+    try:
+        with open("/proc/mounts", "r") as ifp:
+            for line in ifp:
+                procLines = line.rstrip('\n').split()
+                if procLines[1] == mountPoint:
+                    return procLines[0]
+    except EnvironmentError:
+        pass
+    return None
+
+def getDiskData(BBDirs, configuration):
+
+    """Prepare disk data for disk space monitor"""
+
+    # Save the device IDs, need the ID to be unique (the dictionary's key is
+    # unique), so that when more than one directory is located on the same
+    # device, we just monitor it once
+    devDict = {}
+    for pathSpaceInode in BBDirs.split():
+        # The input format is: "dir,space,inode", dir is a must, space
+        # and inode are optional
+        pathSpaceInodeRe = re.match('([^,]*),([^,]*),([^,]*),?(.*)', pathSpaceInode)
+        if not pathSpaceInodeRe:
+            printErr("Invalid value in BB_DISKMON_DIRS: %s" % pathSpaceInode)
+            return None
+
+        action = pathSpaceInodeRe.group(1)
+        if action not in ("ABORT", "STOPTASKS", "WARN"):
+            printErr("Unknown disk space monitor action: %s" % action)
+            return None
+
+        path = os.path.realpath(pathSpaceInodeRe.group(2))
+        if not path:
+            printErr("Invalid path value in BB_DISKMON_DIRS: %s" % pathSpaceInode)
+            return None
+
+        # The disk space or inode is optional, but it should have a correct
+        # value once it is specified
+        minSpace = pathSpaceInodeRe.group(3)
+        if minSpace:
+            minSpace = convertGMK(minSpace)
+            if not minSpace:
+                printErr("Invalid disk space value in BB_DISKMON_DIRS: %s" % pathSpaceInodeRe.group(3))
+                return None
+        else:
+            # None means that it is not specified
+            minSpace = None
+
+        minInode = pathSpaceInodeRe.group(4)
+        if minInode:
+            minInode = convertGMK(minInode)
+            if not minInode:
+                printErr("Invalid inode value in BB_DISKMON_DIRS: %s" % pathSpaceInodeRe.group(4))
+                return None
+        else:
+            # None means that it is not specified
+            minInode = None
+
+        if minSpace is None and minInode is None:
+            printErr("No disk space or inode value in found BB_DISKMON_DIRS: %s" % pathSpaceInode)
+            return None
+        # mkdir for the directory since it may not exist, for example the
+        # DL_DIR may not exist at the very beginning
+        if not os.path.exists(path):
+            bb.utils.mkdirhier(path)
+        dev = getMountedDev(path)
+        # Use path/action as the key
+        devDict[os.path.join(path, action)] = [dev, minSpace, minInode]
+
+    return devDict
+
+def getInterval(configuration):
+
+    """ Get the disk space interval """
+
+    # The default value is 50M and 5K.
+    spaceDefault = 50 * 1024 * 1024
+    inodeDefault = 5 * 1024
+
+    interval = configuration.getVar("BB_DISKMON_WARNINTERVAL", True)
+    if not interval:
+        return spaceDefault, inodeDefault
+    else:
+        # The disk space or inode interval is optional, but it should
+        # have a correct value once it is specified
+        intervalRe = re.match('([^,]*),?\s*(.*)', interval)
+        if intervalRe:
+            intervalSpace = intervalRe.group(1)
+            if intervalSpace:
+                intervalSpace = convertGMK(intervalSpace)
+                if not intervalSpace:
+                    printErr("Invalid disk space interval value in BB_DISKMON_WARNINTERVAL: %s" % intervalRe.group(1))
+                    return None, None
+            else:
+                intervalSpace = spaceDefault
+            intervalInode = intervalRe.group(2)
+            if intervalInode:
+                intervalInode = convertGMK(intervalInode)
+                if not intervalInode:
+                    printErr("Invalid disk inode interval value in BB_DISKMON_WARNINTERVAL: %s" % intervalRe.group(2))
+                    return None, None
+            else:
+                intervalInode = inodeDefault
+            return intervalSpace, intervalInode
+        else:
+            printErr("Invalid interval value in BB_DISKMON_WARNINTERVAL: %s" % interval)
+            return None, None
+
+class diskMonitor:
+
+    """Prepare the disk space monitor data"""
+
+    def __init__(self, configuration):
+
+        self.enableMonitor = False
+        self.configuration = configuration
+
+        BBDirs = configuration.getVar("BB_DISKMON_DIRS", True) or None
+        if BBDirs:
+            self.devDict = getDiskData(BBDirs, configuration)
+            if self.devDict:
+                self.spaceInterval, self.inodeInterval = getInterval(configuration)
+                if self.spaceInterval and self.inodeInterval:
+                    self.enableMonitor = True
+                    # These are for saving the previous disk free space and inode, we
+                    # use them to avoid printing too many warning messages
+                    self.preFreeS = {}
+                    self.preFreeI = {}
+                    # This is for STOPTASKS and ABORT, to avoid printing the message
+                    # repeatedly while waiting for the tasks to finish
+                    self.checked = {}
+                    for k in self.devDict:
+                        self.preFreeS[k] = 0
+                        self.preFreeI[k] = 0
+                        self.checked[k] = False
+                    if self.spaceInterval is None and self.inodeInterval is None:
+                        self.enableMonitor = False
+
+    def check(self, rq):
+
+        """ Take action for the monitor """
+
+        if self.enableMonitor:
+            for k in self.devDict:
+                path = os.path.dirname(k)
+                action = os.path.basename(k)
+                dev = self.devDict[k][0]
+                minSpace = self.devDict[k][1]
+                minInode = self.devDict[k][2]
+
+                st = os.statvfs(path)
+
+                # The free space, float point number
+                freeSpace = st.f_bavail * st.f_frsize
+
+                if minSpace and freeSpace < minSpace:
+                    # Always show warning, the self.checked would always be False if the action is WARN
+                    if self.preFreeS[k] == 0 or self.preFreeS[k] - freeSpace > self.spaceInterval and not self.checked[k]:
+                        logger.warn("The free space of %s (%s) is running low (%.3fGB left)" % \
+                                (path, dev, freeSpace / 1024 / 1024 / 1024.0))
+                        self.preFreeS[k] = freeSpace
+
+                    if action == "STOPTASKS" and not self.checked[k]:
+                        logger.error("No new tasks can be executed since the disk space monitor action is \"STOPTASKS\"!")
+                        self.checked[k] = True
+                        rq.finish_runqueue(False)
+                        bb.event.fire(bb.event.DiskFull(dev, 'disk', freeSpace, path), self.configuration)
+                    elif action == "ABORT" and not self.checked[k]:
+                        logger.error("Immediately abort since the disk space monitor action is \"ABORT\"!")
+                        self.checked[k] = True
+                        rq.finish_runqueue(True)
+                        bb.event.fire(bb.event.DiskFull(dev, 'disk', freeSpace, path), self.configuration)
+
+                # The free inodes, float point number
+                freeInode = st.f_favail
+
+                if minInode and freeInode < minInode:
+                    # Some filesystems use dynamic inodes so can't run out
+                    # (e.g. btrfs). This is reported by the inode count being 0.
+                    if st.f_files == 0:
+                        self.devDict[k][2] = None
+                        continue
+                    # Always show warning, the self.checked would always be False if the action is WARN
+                    if self.preFreeI[k] == 0 or self.preFreeI[k] - freeInode > self.inodeInterval and not self.checked[k]:
+                        logger.warn("The free inode of %s (%s) is running low (%.3fK left)" % \
+                                (path, dev, freeInode / 1024.0))
+                        self.preFreeI[k] = freeInode
+
+                    if action  == "STOPTASKS" and not self.checked[k]:
+                        logger.error("No new tasks can be executed since the disk space monitor action is \"STOPTASKS\"!")
+                        self.checked[k] = True
+                        rq.finish_runqueue(False)
+                        bb.event.fire(bb.event.DiskFull(dev, 'inode', freeInode, path), self.configuration)
+                    elif action  == "ABORT" and not self.checked[k]:
+                        logger.error("Immediately abort since the disk space monitor action is \"ABORT\"!")
+                        self.checked[k] = True
+                        rq.finish_runqueue(True)
+                        bb.event.fire(bb.event.DiskFull(dev, 'inode', freeInode, path), self.configuration)
+        return

bitbake/lib/bb/msg.py

@@ -0,0 +1,199 @@
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+"""
+BitBake 'msg' implementation
+
+Message handling infrastructure for bitbake
+
+"""
+
+# Copyright (C) 2006        Richard Purdie
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import sys
+import copy
+import logging
+import collections
+from itertools import groupby
+import warnings
+import bb
+import bb.event
+
+class BBLogFormatter(logging.Formatter):
+    """Formatter which ensures that our 'plain' messages (logging.INFO + 1) are used as is"""
+
+    DEBUG3 = logging.DEBUG - 2
+    DEBUG2 = logging.DEBUG - 1
+    DEBUG = logging.DEBUG
+    VERBOSE = logging.INFO - 1
+    NOTE = logging.INFO
+    PLAIN = logging.INFO + 1
+    ERROR = logging.ERROR
+    WARNING = logging.WARNING
+    CRITICAL = logging.CRITICAL
+
+    levelnames = {
+        DEBUG3   : 'DEBUG',
+        DEBUG2   : 'DEBUG',
+        DEBUG   : 'DEBUG',
+        VERBOSE: 'NOTE',
+        NOTE    : 'NOTE',
+        PLAIN  : '',
+        WARNING : 'WARNING',
+        ERROR   : 'ERROR',
+        CRITICAL: 'ERROR',
+    }
+
+    color_enabled = False
+    BASECOLOR, BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE = range(29,38)
+
+    COLORS = {
+        DEBUG3  : CYAN,
+        DEBUG2  : CYAN,
+        DEBUG   : CYAN,
+        VERBOSE : BASECOLOR,
+        NOTE    : BASECOLOR,
+        PLAIN   : BASECOLOR,
+        WARNING : YELLOW,
+        ERROR   : RED,
+        CRITICAL: RED,
+    }
+
+    BLD = '\033[1;%dm'
+    STD = '\033[%dm'
+    RST = '\033[0m'
+
+    def getLevelName(self, levelno):
+        try:
+            return self.levelnames[levelno]
+        except KeyError:
+            self.levelnames[levelno] = value = 'Level %d' % levelno
+            return value
+
+    def format(self, record):
+        record.levelname = self.getLevelName(record.levelno)
+        if record.levelno == self.PLAIN:
+            msg = record.getMessage()
+        else:
+            if self.color_enabled:
+                record = self.colorize(record)
+            msg = logging.Formatter.format(self, record)
+
+        if hasattr(record, 'bb_exc_info'):
+            etype, value, tb = record.bb_exc_info
+            formatted = bb.exceptions.format_exception(etype, value, tb, limit=5)
+            msg += '\n' + ''.join(formatted)
+        return msg
+
+    def colorize(self, record):
+        color = self.COLORS[record.levelno]
+        if self.color_enabled and color is not None:
+            record = copy.copy(record)
+            record.levelname = "".join([self.BLD % color, record.levelname, self.RST])
+            record.msg = "".join([self.STD % color, record.msg, self.RST])
+        return record
+
+    def enable_color(self):
+        self.color_enabled = True
+
+class BBLogFilter(object):
+    def __init__(self, handler, level, debug_domains):
+        self.stdlevel = level
+        self.debug_domains = debug_domains
+        loglevel = level
+        for domain in debug_domains:
+            if debug_domains[domain] < loglevel:
+                loglevel = debug_domains[domain]
+        handler.setLevel(loglevel)
+        handler.addFilter(self)
+
+    def filter(self, record):
+        if record.levelno >= self.stdlevel:
+            return True
+        if record.name in self.debug_domains and record.levelno >= self.debug_domains[record.name]:
+            return True
+        return False
+
+class BBLogFilterStdErr(BBLogFilter):
+    def filter(self, record):
+        if not BBLogFilter.filter(self, record):
+            return False
+        if record.levelno >= logging.ERROR:
+            return True
+        return False
+
+class BBLogFilterStdOut(BBLogFilter):
+    def filter(self, record):
+        if not BBLogFilter.filter(self, record):
+            return False
+        if record.levelno < logging.ERROR:
+            return True
+        return False
+
+# Message control functions
+#
+
+loggerDefaultDebugLevel = 0
+loggerDefaultVerbose = False
+loggerVerboseLogs = False
+loggerDefaultDomains = []
+
+def init_msgconfig(verbose, debug, debug_domains=None):
+    """
+    Set default verbosity and debug levels config the logger
+    """
+    bb.msg.loggerDefaultDebugLevel = debug
+    bb.msg.loggerDefaultVerbose = verbose
+    if verbose:
+        bb.msg.loggerVerboseLogs = True
+    if debug_domains:
+        bb.msg.loggerDefaultDomains = debug_domains
+    else:
+        bb.msg.loggerDefaultDomains = []
+
+def constructLogOptions():
+    debug = loggerDefaultDebugLevel
+    verbose = loggerDefaultVerbose
+    domains = loggerDefaultDomains
+
+    if debug:
+        level = BBLogFormatter.DEBUG - debug + 1
+    elif verbose:
+        level = BBLogFormatter.VERBOSE
+    else:
+        level = BBLogFormatter.NOTE
+
+    debug_domains = {}
+    for (domainarg, iterator) in groupby(domains):
+        dlevel = len(tuple(iterator))
+        debug_domains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
+    return level, debug_domains
+
+def addDefaultlogFilter(handler, cls = BBLogFilter):
+    level, debug_domains = constructLogOptions()
+
+    cls(handler, level, debug_domains)
+
+#
+# Message handling functions
+#
+
+def fatal(msgdomain, msg):
+    if msgdomain:
+        logger = logging.getLogger("BitBake.%s" % msgdomain)
+    else:
+        logger = logging.getLogger("BitBake")
+    logger.critical(msg)
+    sys.exit(1)

bitbake/lib/bb/namedtuple_with_abc.py

@@ -0,0 +1,255 @@
+# http://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-class-mix-in-for-named/
+#!/usr/bin/env python
+# Copyright (c) 2011 Jan Kaliszewski (zuo). Available under the MIT License.
+
+"""
+namedtuple_with_abc.py:
+* named tuple mix-in + ABC (abstract base class) recipe,
+* works under Python 2.6, 2.7 as well as 3.x.
+
+Import this module to patch collections.namedtuple() factory function
+-- enriching it with the 'abc' attribute (an abstract base class + mix-in
+for named tuples) and decorating it with a wrapper that registers each
+newly created named tuple as a subclass of namedtuple.abc.
+
+How to import:
+    import collections, namedtuple_with_abc
+or:
+    import namedtuple_with_abc
+    from collections import namedtuple
+    # ^ in this variant you must import namedtuple function
+    #   *after* importing namedtuple_with_abc module
+or simply:
+    from namedtuple_with_abc import namedtuple
+
+Simple usage example:
+    class Credentials(namedtuple.abc):
+        _fields = 'username password'
+        def __str__(self):
+            return ('{0.__class__.__name__}'
+                    '(username={0.username}, password=...)'.format(self))
+    print(Credentials("alice", "Alice's password"))
+
+For more advanced examples -- see below the "if __name__ == '__main__':".
+"""
+
+import collections
+from abc import ABCMeta, abstractproperty
+from functools import wraps
+from sys import version_info
+
+__all__ = ('namedtuple',)
+_namedtuple = collections.namedtuple
+
+
+class _NamedTupleABCMeta(ABCMeta):
+    '''The metaclass for the abstract base class + mix-in for named tuples.'''
+    def __new__(mcls, name, bases, namespace):
+        fields = namespace.get('_fields')
+        for base in bases:
+            if fields is not None:
+                break
+            fields = getattr(base, '_fields', None)
+        if not isinstance(fields, abstractproperty):
+            basetuple = _namedtuple(name, fields)
+            bases = (basetuple,) + bases
+            namespace.pop('_fields', None)
+            namespace.setdefault('__doc__', basetuple.__doc__)
+            namespace.setdefault('__slots__', ())
+        return ABCMeta.__new__(mcls, name, bases, namespace)
+
+
+exec(
+    # Python 2.x metaclass declaration syntax
+    """class _NamedTupleABC(object):
+        '''The abstract base class + mix-in for named tuples.'''
+        __metaclass__ = _NamedTupleABCMeta
+        _fields = abstractproperty()""" if version_info[0] < 3 else
+    # Python 3.x metaclass declaration syntax
+    """class _NamedTupleABC(metaclass=_NamedTupleABCMeta):
+        '''The abstract base class + mix-in for named tuples.'''
+        _fields = abstractproperty()"""
+)
+
+
+_namedtuple.abc = _NamedTupleABC
+#_NamedTupleABC.register(type(version_info))  # (and similar, in the future...)
+
+@wraps(_namedtuple)
+def namedtuple(*args, **kwargs):
+    '''Named tuple factory with namedtuple.abc subclass registration.'''
+    cls = _namedtuple(*args, **kwargs)
+    _NamedTupleABC.register(cls)
+    return cls
+
+collections.namedtuple = namedtuple
+
+
+
+
+if __name__ == '__main__':
+
+    '''Examples and explanations'''
+
+    # Simple usage
+
+    class MyRecord(namedtuple.abc):
+        _fields = 'x y z'  # such form will be transformed into ('x', 'y', 'z')
+        def _my_custom_method(self):
+            return list(self._asdict().items())
+    # (the '_fields' attribute belongs to the named tuple public API anyway)
+
+    rec = MyRecord(1, 2, 3)
+    print(rec)
+    print(rec._my_custom_method())
+    print(rec._replace(y=222))
+    print(rec._replace(y=222)._my_custom_method())
+
+    # Custom abstract classes...
+
+    class MyAbstractRecord(namedtuple.abc):
+        def _my_custom_method(self):
+            return list(self._asdict().items())
+
+    try:
+        MyAbstractRecord()  # (abstract classes cannot be instantiated)
+    except TypeError as exc:
+        print(exc)
+
+    class AnotherAbstractRecord(MyAbstractRecord):
+        def __str__(self):
+            return '<<<{0}>>>'.format(super(AnotherAbstractRecord,
+                                            self).__str__())
+
+    # ...and their non-abstract subclasses
+
+    class MyRecord2(MyAbstractRecord):
+        _fields = 'a, b'
+
+    class MyRecord3(AnotherAbstractRecord):
+        _fields = 'p', 'q', 'r'
+
+    rec2 = MyRecord2('foo', 'bar')
+    print(rec2)
+    print(rec2._my_custom_method())
+    print(rec2._replace(b=222))
+    print(rec2._replace(b=222)._my_custom_method())
+
+    rec3 = MyRecord3('foo', 'bar', 'baz')
+    print(rec3)
+    print(rec3._my_custom_method())
+    print(rec3._replace(q=222))
+    print(rec3._replace(q=222)._my_custom_method())
+
+   # You can also subclass non-abstract ones...
+
+    class MyRecord33(MyRecord3):
+        def __str__(self):
+            return '< {0!r}, ..., {0!r} >'.format(self.p, self.r)
+
+    rec33 = MyRecord33('foo', 'bar', 'baz')
+    print(rec33)
+    print(rec33._my_custom_method())
+    print(rec33._replace(q=222))
+    print(rec33._replace(q=222)._my_custom_method())
+
+    # ...and even override the magic '_fields' attribute again
+
+    class MyRecord345(MyRecord3):
+        _fields = 'e f g h i j k'
+
+    rec345 = MyRecord345(1, 2, 3, 4, 3, 2, 1)
+    print(rec345)
+    print(rec345._my_custom_method())
+    print(rec345._replace(f=222))
+    print(rec345._replace(f=222)._my_custom_method())
+
+    # Mixing-in some other classes is also possible:
+
+    class MyMixIn(object):
+        def method(self):
+            return "MyMixIn.method() called"
+        def _my_custom_method(self):
+            return "MyMixIn._my_custom_method() called"
+        def count(self, item):
+            return "MyMixIn.count({0}) called".format(item)
+        def _asdict(self):  # (cannot override a namedtuple method, see below)
+            return "MyMixIn._asdict() called"
+
+    class MyRecord4(MyRecord33, MyMixIn):  # mix-in on the right
+        _fields = 'j k l x'
+
+    class MyRecord5(MyMixIn, MyRecord33):  # mix-in on the left
+        _fields = 'j k l x y'
+
+    rec4 = MyRecord4(1, 2, 3, 2)
+    print(rec4)
+    print(rec4.method())
+    print(rec4._my_custom_method())  # MyRecord33's
+    print(rec4.count(2))  # tuple's
+    print(rec4._replace(k=222))
+    print(rec4._replace(k=222).method())
+    print(rec4._replace(k=222)._my_custom_method())  # MyRecord33's
+    print(rec4._replace(k=222).count(8))  # tuple's
+
+    rec5 = MyRecord5(1, 2, 3, 2, 1)
+    print(rec5)
+    print(rec5.method())
+    print(rec5._my_custom_method())  # MyMixIn's
+    print(rec5.count(2))  # MyMixIn's
+    print(rec5._replace(k=222))
+    print(rec5._replace(k=222).method())
+    print(rec5._replace(k=222)._my_custom_method())  # MyMixIn's
+    print(rec5._replace(k=222).count(2))  # MyMixIn's
+
+    # Note that behavior: the standard namedtuple methods cannot be
+    # overridden by a foreign mix-in -- even if the mix-in is declared
+    # as the leftmost base class (but, obviously, you can override them
+    # in the defined class or its subclasses):
+
+    print(rec4._asdict())  # (returns a dict, not "MyMixIn._asdict() called")
+    print(rec5._asdict())  # (returns a dict, not "MyMixIn._asdict() called")
+
+    class MyRecord6(MyRecord33):
+        _fields = 'j k l x y z'
+        def _asdict(self):
+            return "MyRecord6._asdict() called"
+    rec6 = MyRecord6(1, 2, 3, 1, 2, 3)
+    print(rec6._asdict())  # (this returns "MyRecord6._asdict() called")
+
+    # All that record classes are real subclasses of namedtuple.abc:
+
+    assert issubclass(MyRecord, namedtuple.abc)
+    assert issubclass(MyAbstractRecord, namedtuple.abc)
+    assert issubclass(AnotherAbstractRecord, namedtuple.abc)
+    assert issubclass(MyRecord2, namedtuple.abc)
+    assert issubclass(MyRecord3, namedtuple.abc)
+    assert issubclass(MyRecord33, namedtuple.abc)
+    assert issubclass(MyRecord345, namedtuple.abc)
+    assert issubclass(MyRecord4, namedtuple.abc)
+    assert issubclass(MyRecord5, namedtuple.abc)
+    assert issubclass(MyRecord6, namedtuple.abc)
+
+    # ...but abstract ones are not subclasses of tuple
+    # (and this is what you probably want):
+
+    assert not issubclass(MyAbstractRecord, tuple)
+    assert not issubclass(AnotherAbstractRecord, tuple)
+
+    assert issubclass(MyRecord, tuple)
+    assert issubclass(MyRecord2, tuple)
+    assert issubclass(MyRecord3, tuple)
+    assert issubclass(MyRecord33, tuple)
+    assert issubclass(MyRecord345, tuple)
+    assert issubclass(MyRecord4, tuple)
+    assert issubclass(MyRecord5, tuple)
+    assert issubclass(MyRecord6, tuple)
+
+    # Named tuple classes created with namedtuple() factory function
+    # (in the "traditional" way) are registered as "virtual" subclasses
+    # of namedtuple.abc:
+
+    MyTuple = namedtuple('MyTuple', 'a b c')
+    mt = MyTuple(1, 2, 3)
+    assert issubclass(MyTuple, namedtuple.abc)
+    assert isinstance(mt, namedtuple.abc)