x11-common: Fix unusable serial console

The serial console port is basicly unusable in images containing X.
Login works fine, but at the shell prompt only one out of N input
characters (N usually between 2 and 10) gets through to the shell.

dbus-launch (running as "dbus-launch --sh-syntax --exit-with-session")
is also reading from /dev/console and "eating" the missing characters.

As soon as I stop the Xserver ("sh /etc/init.d/xserver-nodm stop")
the serial console starts wroking fine (because dbus-launch is not
running any more).

This patch addresses the problem.

Signed-off-by: Wolfgang Denk <wd@denx.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

.gitignore

@@ -1,20 +1,26 @@
 *.pyc
 *.pyo
-build*/conf/local.conf
-build*/conf/bblayers.conf
-build*/downloads
-build*/tmp/
-build*/sstate-cache
-build*/pyshtables.py
+build/conf/local.conf
+build/conf/bblayers.conf
+build/tmp/
+build/sstate-cache
+build/pyshtables.py
 pstage/
-scripts/oe-git-proxy-socks
+scripts/poky-git-proxy-socks
 sources/
-meta-*
-!meta-skeleton
-!meta-demoapps
+meta-darwin
+meta-maemo
+meta-extras
+meta-m2
+meta-prvt*
+poky-autobuilder*
 *.swp
 *.orig
 *.rej
 *~
-
+documentation/poky-ref-manual/poky-ref-manual.html
+documentation/poky-ref-manual/poky-ref-manual.pdf
+documentation/poky-ref-manual/poky-ref-manual.tgz
+documentation/poky-ref-manual/bsp-guide.html
+documentation/poky-ref-manual/bsp-guide.pdf
 

CHANGELOG

@@ -0,0 +1,220 @@
+commit fd7a07b3a2153826bedda2ef76b9a33ab2791680
+Author: Scott Garman <scott.a.garman@intel.com>
+Date:   Fri Jan 21 14:15:05 2011 -0800
+
+    poky-extract-sdk: allow relative paths for extract-dir
+    
+    psuedo needs a full path to its pid file, so convert
+    relative extract-dir paths to full ones.
+    
+    The symptom of this bug is receiving the following error:
+    
+    pseudo: Couldn't open relative/path/to/var/pseudo/pseudo.pid: No such file or directory
+    
+    This fixes [BUGID #670]
+    
+    Signed-off-by: Scott Garman <scott.a.garman@intel.com>
+
+commit 01bc47f4d47df3276b4b6c2583bcddd834fd5050
+Author: Beth Flanagan <elizabeth.flanagan@intel.com>
+Date:   Wed Nov 3 17:20:00 2010 -0700
+
+    quilt: Fixed configure test for patch --version.
+    
+    OpenSuSE 11.3 uses GNU patch 2.6.1.81-5b68 which breaks quilt's
+    configure test for patch version.
+    
+    Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
+
+commit 12a3d41a24db79ae6c0491defffcf4f4753001cf
+Author: Richard Purdie <richard.purdie@linuxfoundation.org>
+Date:   Fri Jan 14 11:57:18 2011 +0000
+
+    image.bbclass: Use the dedicated BB_WORKERCONTEXT, not bitbake internals to detect context
+    
+    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+
+commit ce4f8356796bc797d9156ed252a4ed638a2150d5
+Author: Richard Purdie <rpurdie@linux.intel.com>
+Date:   Wed Dec 15 23:22:16 2010 +0000
+
+    scripts/poky-qemu: Improve tmp layout assumption
+    
+    If someone has changed TMPDIR in local.conf to a non-standard location, the
+    poky-qemu script currently doesn't handle this and assumes if BUILDDIR is set,
+    $BUILDDIR/tmp will exist.
+    
+    Its simple to check if this exists and if not, to ask bitbake where the
+    directory is so this patch changes the code to do that.
+    
+    Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
+
+commit 54f08d23cd7d0de6aec31f4764389ff4dab2990d
+Author: Scott Garman <scott.a.garman@intel.com>
+Date:   Tue Dec 7 20:59:06 2010 -0800
+
+    Make poky-qemu and related scripts work with arbitrary SDK locations
+    
+    * No longer assume SDK toolchains are installed in /opt/poky
+    * [BUGFIX #568] where specifying paths to both the kernel and fs
+      image caused an error due to POKY_NATIVE_SYSROOT never being
+      set, triggering failure of poky-qemu-ifup/ifdown
+    * Cosmetic improvements to usage() functions by using basename
+    
+    Signed-off-by: Scott Garman <scott.a.garman@intel.com>
+
+commit 8a3d0f375ce416ada1a5443e4a8e467504001beb
+Author: Scott Garman <scott.a.garman@intel.com>
+Date:   Fri Nov 12 16:31:13 2010 -0800
+
+    poky-qemu: Fix issues when running Yocto 0.9 release images
+    
+    This fixes two bugs with poky-qemu when it is run from a
+    standalone meta-toolchain setup.
+    
+    [BUGFIX #535] and [BUGFIX #536]
+    
+    Signed-off-by: Scott Garman <scott.a.garman@intel.com>
+
+commit 0c2003f13434c77f901a976523478d37d8aadb48
+Author: Paul Eggleton <paul.eggleton@linux.intel.com>
+Date:   Thu Dec 16 10:29:50 2010 +0000
+
+    openssl: restore -Wall flag
+    
+    The -Wall flag was unintentionally removed from the end of the CFLAG var in
+    089612794d4d8d9c79bd2a4365d6df78371f7f40 by me. This patch puts it back in.
+    
+    Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
+    Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
+
+commit 6e71b0a012f0676c06b7b4788d932f320fca0b74
+Author: Joshua Lock <josh@linux.intel.com>
+Date:   Wed Dec 15 14:31:21 2010 +0000
+
+    web-webkit: fix for make 3.82
+    
+    Signed-off-by: Joshua Lock <josh@linux.intel.com>
+
+commit 4b5c1c053000d297956f08949ffde7454ee33c5d
+Author: Joshua Lock <josh@linux.intel.com>
+Date:   Wed Dec 15 13:42:15 2010 +0000
+
+    contacts: fix for make 3.82
+    
+    Signed-off-by: Joshua Lock <josh@linux.intel.com>
+
+commit 171e709ae6f4b1a7640bf393f57aa787648cdc0f
+Author: Joshua Lock <josh@linux.intel.com>
+Date:   Wed Dec 15 12:58:09 2010 +0000
+
+    dates: fix for Make 3.82
+    
+    Signed-off-by: Joshua Lock <josh@linux.intel.com>
+
+commit a8b8557e4cb34b594bb620eb276bcaf7a8e0a8e3
+Author: Joshua Lock <josh@linux.intel.com>
+Date:   Wed Dec 15 12:27:52 2010 +0000
+
+    owl-video-widget: fix Makefile for super strict make 3.82
+    
+    Signed-off-by: Joshua Lock <josh@linux.intel.com>
+
+commit 399e6b8008cb0b8cc0b75efd48dd821a6cf5a8a8
+Author: Joshua Lock <josh@linux.intel.com>
+Date:   Tue Dec 14 18:29:43 2010 +0000
+
+    libowl-av: fix for Make 3.82
+    
+    Signed-off-by: Joshua Lock <josh@linux.intel.com>
+
+commit 290280b332570ec73301f76765b1c5f2de20a9fd
+Author: Joshua Lock <josh@linux.intel.com>
+Date:   Tue Dec 14 17:56:53 2010 +0000
+
+    gst-plugins: fix for make 3.82
+    
+    Signed-off-by: Joshua Lock <josh@linux.intel.com>
+
+commit 9e11fbf9048b17526ca8160d82b69f386595c9a7
+Author: Joshua Lock <josh@linux.intel.com>
+Date:   Tue Dec 14 15:39:42 2010 +0000
+
+    gstreamer: fix to comply with make 3.82's stricter parser
+    
+    Signed-off-by: Joshua Lock <josh@linux.intel.com>
+
+commit 0f8244faba5c36c0580081c112ea27ce683af99b
+Author: Joshua Lock <josh@linux.intel.com>
+Date:   Tue Dec 14 12:49:13 2010 +0000
+
+    linux-libc-headers: fix for Make 3.82
+    
+    Fix the kernel Makefile for use with Make 3.82 by splitting mixed implicit and
+    normal rules into separate rules.
+    
+    Signed-off-by: Joshua Lock <josh@linux.intel.com>
+
+commit 0cc23a86562d0ce1e236ceb4a56a8f19d400192f
+Author: Joshua Lock <josh@linux.intel.com>
+Date:   Tue Dec 14 12:21:33 2010 +0000
+
+    busybox: additional fixes for Make 3.82
+    
+    There where still some mixed implicit and normal rules in the Busybox Makefile,
+    Update our existing make-382.patch to split these into separate rules.
+    
+    Signed-off-by: Joshua Lock <josh@linux.intel.com>
+
+commit 30c39cc97c384134661300e107d7a81f257f8034
+Author: Joshua Lock <josh@linux.intel.com>
+Date:   Fri Nov 12 16:36:54 2010 +0000
+
+    procps: fix for build against make 3.82
+    
+    Signed-off-by: Joshua Lock <josh@linux.intel.com>
+
+commit 261ca885962ba9606bcad4c5415927a79fdd7b96
+Author: Joshua Lock <josh@linux.intel.com>
+Date:   Tue Nov 9 12:18:14 2010 +0000
+
+    busybox: import upstream patch for make 3.82
+    
+    Signed-off-by: Joshua Lock <josh@linux.intel.com>
+
+commit 72ddd5c20246a5d5b1752b58a61ef75b4c39cc40
+Author: Joshua Lock <josh@linux.intel.com>
+Date:   Tue Nov 9 12:14:28 2010 +0000
+
+    eglibc: fix build of eglibc-initial for make 3.82
+    
+    Make 3.82, as shipped with Fedora 14, fixes some holes in the parser which in
+    turn breaks behaviour of some Makefiles. Most notably eglibc's.
+    
+    Signed-off-by: Joshua Lock <josh@linux.intel.com>
+
+commit 6026999e81042a7f6560f9bce04390865509b235
+Author: Paul Eggleton <paul.eggleton@intel.com>
+Date:   Fri Nov 19 15:03:32 2010 +0000
+
+    qemu: fix failure to find zlib header files during configure
+    
+    Corrects problems during configure of qemu-native due to the BUILD_CFLAGS
+    not being included when attempting to compile the test program for zlib
+    within the configure script.
+    
+    Signed-off-by: Paul Eggleton <paul.eggleton@intel.com>
+    Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
+
+commit c5ab4d56f97a0e45b124d40c9f536541be04c201
+Author: Paul Eggleton <paul.eggleton@intel.com>
+Date:   Wed Nov 17 11:37:47 2010 +0000
+
+    openssl-native: disable execstack flag to prevent problems with SELinux
+    
+    The execstack flag gets set on libcrypto.so by default which causes SELinux
+    to prevent it from being loaded on systems using SELinux, which includes
+    Fedora. This patch disables the execstack flag. (Note: Red Hat do this in
+    their openssl packaging.)
+    
+    Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>

NOTES

@@ -0,0 +1,70 @@
+Name: Laverne
+Version: 4.0.1
+Built from Revision: fd7a07b3a2153826bedda2ef76b9a33ab2791680
+Build Date: Jan 26 2011
+Builder: autobuilder.pokylinux.org 
+
+The Laverne 4.0.1 Release ensures you can use Poky Laverne on systems running 
+Fedora 14 and Opensuse 11.3, fixes issues with the poky-qemu script, and fixes 
+several other bugs. For the full changelog for Laverne 4.0.1 please read 
+CHANGELOG.
+
+Following are descriptions of fixes and known issues.
+
+Fixes
+------------------------
+
+    * Make 3.82, as shipped with Fedora 14, included parser bug fixes that 
+    resulted in a much stricter parser. As a result, the Makefiles could not be 
+    parsed for many of the software versions shipped with Laverne. The Makefiles
+    in the following recipes were fixed:
+
+          o eglibc
+          o busybox
+          o procps
+          o linux-libc-headers
+          o gstreamer
+          o gst-plugins
+          o libowl-av
+          o owl-video-widget
+          o dates
+          o contacts
+          o web-webkit 
+
+    * The ability to build openssl-native on a system that has SELINUX enabled 
+    was restored. (We disabled the execstack flag at compile time.) 
+
+    * A host-intrusion issue caused by a failure in QEMU to find zlib headers 
+    during configure was fixed. The issue was causing qemu-native to use the 
+    system zlib if it was present. If the system zlib was not present the build 
+    would fail. 
+
+    * Stability and usability enhancements, which included handling relative 
+    filesystem paths, were made to poky-qemu scripts. 
+
+    * The run-time remapping of package names when adding extra packages to an 
+    image via the IMAGE_INSTALL mechanism were fixed. 
+
+    * The configure test in quilt for GNU patch was fixed to that it correctly 
+    detects the version. 
+
+Known Issues
+------------------------
+
+    * The mpc3815e-rbd and routerstationpro machines were untested and not a 
+    part of the official Laverne 4.0 release. These machines are still unusable 
+    for this Laverne 4.0.1 release.
+          o mpx3815e-rdb will not boot due to a kernel/uboot issue Bug #685
+          o routerstation will not boot (by default) due to incorrect boot 
+          parameters Bug #681
+          o routerstationpro debug messages related to the ethernet driver print 
+          during boot Bug #679 
+    * Shutdown/poweroff on qemuarm does not cleanly halt the virtual machine. 
+    To workaround this issue use the reboot command. Using this command avoids 
+    a "power-cycle" and instead cleanly shuts down the VM Bug #684
+    * Two "Connection Manager" icons appear in the Sato UI. This duplication has
+    been fixed in master. Note that you can use either icon to launch the 
+    connectivity UI. Bug #683
+    * The on-screen keyboard incorrectly launches in the qemumips machine. This 
+    issue is due to a mis-configured formfactor file Bug #682 
+

README

@@ -1,25 +1,15 @@
 Poky
 ====
 
-Poky is an integration of various components to form a complete prepackaged
-build system and development environment. It features support for building
-customised embedded device style images. There are reference demo images
-featuring a X11/Matchbox/GTK themed UI called Sato. The system supports
-cross-architecture application development using QEMU emulation and a
-standalone toolchain and SDK with IDE integration.
+Poky platform builder is a combined cross build system and development 
+environment. It features support for building X11/Matchbox/GTK based 
+filesystem images for various embedded devices and boards. It also 
+supports cross-architecture application development using QEMU emulation
+and a standalone toolchain and SDK with IDE integration.
+
+Poky has an extensive handbook, the source of which is contained in
+the handbook directory. For compiled HTML or pdf versions of this, 
+see the Poky website http://pokylinux.org.
 
 Additional information on the specifics of hardware that Poky supports
-is available in README.hardware. Further hardware support can easily be added
-in the form of layers which extend the systems capabilities in a modular way.
-
-As an integration layer Poky consists of several upstream projects such as 
-BitBake, OpenEmbedded-Core, Yocto documentation and various sources of information 
-e.g. for the hardware support. Poky is in turn a component of the Yocto Project.
-
-The Yocto Project has extensive documentation about the system including a 
-reference manual which can be found at:
-    http://yoctoproject.org/community/documentation
-
-For information about OpenEmbedded see their website:
-    http://www.openembedded.org/
-
+is available in README.hardware.

README.hardware

@@ -1,447 +1,436 @@
-                          Poky Hardware README
-                          ====================
+                     Poky Hardware Reference Guide
+                     =============================
 
 This file gives details about using Poky with different hardware reference
-boards and consumer devices. A full list of target machines can be found by
-looking in the meta/conf/machine/ directory. If in doubt about using Poky with
-your hardware, consult the documentation for your board/device.
+boards and consumer devices. A full list of target machines can be found by 
+looking in the meta/conf/machine/ directory. If in doubt about using Poky with 
+your hardware, consult the documentation for your board/device. To discuss 
+support for further hardware reference boards/devices please contact OpenedHand.
 
-Support for additional devices is normally added by creating BSP layers - for
-more information please see the Yocto Board Support Package (BSP) Developer's
-Guide - documentation source is in documentation/bspguide or download the PDF
-from:
-
-   http://yoctoproject.org/community/documentation
-
-Support for machines other than QEMU may be moved out to separate BSP layers in
-future versions.
-
-
-QEMU Emulation Targets
-======================
+QEMU Emulation Images (qemuarm and qemux86)
+===========================================
 
 To simplify development Poky supports building images to work with the QEMU
-emulator in system emulation mode. Several architectures are currently
-supported:
-
-  * ARM (qemuarm)
-  * x86 (qemux86)
-  * x86-64 (qemux86-64)
-  * PowerPC (qemuppc)
-  * MIPS (qemumips)
-
-Use of the QEMU images is covered in the Poky Reference Manual. The Poky
-MACHINE setting corresponding to the target is given in brackets.
-
+emulator in system emulation mode. Two architectures are currently supported,
+ARM (via qemuarm) and x86 (via qemux86). Use of the QEMU images is covered
+in the Poky Handbook.
 
 Hardware Reference Boards
 =========================
 
-The following boards are supported by Poky's core layer:
+The following boards are supported by Poky:
 
-  * Texas Instruments Beagleboard (beagleboard)
-  * Freescale MPC8315E-RDB (mpc8315e-rdb)
-  * Ubiquiti Networks RouterStation Pro (routerstationpro)
+  * Compulab CM-X270 (cm-x270)
+  * Compulab EM-X270 (em-x270)
+  * FreeScale iMX31ADS (mx31ads)
+  * Marvell PXA3xx Zylonite (zylonite)
+  * Logic iMX31 Lite Kit (mx31litekit)
+  * Phytec phyCORE-iMX31 (mx31phy)
 
-For more information see the board's section below. The Poky MACHINE setting
+For more information see board's section below. The Poky MACHINE setting
 corresponding to the board is given in brackets.
 
-
 Consumer Devices
 ================
 
-The following consumer devices are supported by Poky's core layer:
+The following consumer devices are supported by Poky:
 
-  * Intel Atom based PCs and devices (atom-pc)
+  * FIC Neo1973 GTA01 smartphone (fic-gta01)
+  * HTC Universal (htcuniversal)
+  * Nokia 770/N800/N810 Internet Tablets (nokia770 and nokia800)
+  * Sharp Zaurus SL-C7x0 series (c7x0)
+  * Sharp Zaurus SL-C1000 (akita)
+  * Sharp Zaurus SL-C3x00 series (spitz)
 
-For more information see the device's section below. The Poky MACHINE setting
-corresponding to the device is given in brackets.
+For more information see board's section below. The Poky MACHINE setting
+corresponding to the board is given in brackets.
 
+Poky Boot CD (bootcdx86)
+========================
 
+The Poky boot CD iso images are designed as a demonstration of the Poky 
+environment and to show the versatile image formats Poky can generate. It will
+run on Pentium2 or greater PC style computers. The iso image can be 
+burnt to CD and then booted from.
 
-                      Specific Hardware Documentation
-                      ===============================
 
+                          Hardware Reference Boards
+                          =========================
 
-Intel Atom based PCs and devices (atom-pc)
-==========================================
+Compulab CM-X270 (cm-x270)
+==========================
 
-The atom-pc MACHINE is tested on the following platforms:
+The bootloader on this board doesn't support writing jffs2 images directly to 
+NAND and normally uses a proprietary kernel flash driver. To allow the use of
+jffs2 images, a two stage updating procedure is needed. Firstly, an initramfs
+is booted which contains mtd utilities and this is then used to write the main
+filesystem. 
 
-  o Asus EeePC 901
-  o Acer Aspire One
-  o Toshiba NB305
-  o Intel Embedded Development Board 1-N450 (Black Sand)
+It is assumed the board is connected to a network where a TFTP server is 
+available and that a serial terminal is available to communicate with the 
+bootloader (38400, 8N1). If a DHCP server is available the device will use it
+to obtain an IP address. If not, run:
 
-and is likely to work on many unlisted Atom based devices. The MACHINE type
-supports ethernet, wifi, sound, and i915 graphics by default in addition to
-common PC input devices, busses, and so on.
+  ARMmon > setip dhcp off
+  ARMmon > setip ip 192.168.1.203
+  ARMmon > setip mask 255.255.255.0
 
-Depending on the device, it can boot from a traditional hard-disk, a USB device,
-or over the network. Writing poky generated images to physical media is
-straightforward with a caveat for USB devices. The following examples assume the
-target boot device is /dev/sdb, be sure to verify this and use the correct
-device as the following commands are run as root and are not reversable.
+To reflash the kernel:
 
-USB Device:
-  1. Build a live image. This image type consists of a simple filesystem
-     without a partition table, which is suitable for USB keys, and with the
-     default setup for the atom-pc machine, this image type is built
-     automatically for any image you build. For example:
+  ARMmon > download kernel tftp zimage 192.168.1.202
+  ARMmon > flash kernel
 
-     $ bitbake core-image-minimal
+where zimage is the name of the kernel on the TFTP server and its IP address is 
+192.168.1.202. The names of the files must be all lowercase.
 
-  2. Use the "dd" utility to write the image to the raw block device. For
-     example:
+To reflash the initrd/initramfs:
 
-     # dd if=core-image-minimal-atom-pc.hddimg of=/dev/sdb
+  ARMmon > download ramdisk tftp diskimage 192.168.1.202
+  ARMmon > flash ramdisk
 
-  If the device fails to boot with "Boot error" displayed, it is likely the BIOS
-  cannot understand the physical layout of the disk (or rather it expects a
-  particular layout and cannot handle anything else). There are two possible
-  solutions to this problem:
+where diskimage is the name of the initramfs image (a cpio.gz file).
 
-  1. Change the BIOS USB Device setting to HDD mode. The label will vary by
-     device, but the idea is to force BIOS to read the Cylinder/Head/Sector
-     geometry from the device.
+To boot the initramfs:
 
-  2. Without such an option, the BIOS generally boots the device in USB-ZIP
-     mode.
+  ARMmon > ramdisk on
+  ARMmon > bootos "console=ttyS0,38400 rdinit=/sbin/init"
 
-     a. Configure the USB device for USB-ZIP mode:
-     
-     # mkdiskimage -4 /dev/sdb 0 63 62
+To reflash the main image login to the system as user "root", then run:
 
-     Where 63 and 62 are the head and sector count as reported by fdisk.
-     Remove and reinsert the device to allow the kernel to detect the new
-     partition layout.
+  # ifconfig eth0 192.168.1.203
+  # tftp -g -r mainimage 192.168.1.202
+  # flash_eraseall /dev/mtd1
+  # nandwrite /dev/mtd1 mainimage
 
-     b. Copy the contents of the poky image to the USB-ZIP mode device:
+which configures the network interface with the IP address 192.168.1.203,
+downloads the "mainimage" file from the TFTP server at 192.168.1.202, erases
+the flash and then writes the new image to the flash.
 
-     # mount -o loop core-image-minimal-atom-pc.hddimg  /tmp/image
-     # mount /dev/sdb4 /tmp/usbkey
-     # cp -rf /tmp/image/* /tmp/usbkey
+The main image can then be booted with:
 
-     c. Install the syslinux boot loader:
+  ARMmon > bootos "console=ttyS0,38400 root=/dev/mtdblock1 rootfstype=jffs2"
 
-     # syslinux /dev/sdb4
+Note that the initramfs image is built by poky in a slightly different mode to
+normal since it uses uclibc. To generate this use a command like:
 
-  Install the boot device in the target board and configure the BIOS to boot
-  from it.
+IMAGE_FSTYPES=cpio.gz MACHINE=cm-x270 POKYLIBC=uclibc bitbake poky-image-minimal-mtdutils
 
-  For more details on the USB-ZIP scenario, see the syslinux documentation:
-  http://git.kernel.org/?p=boot/syslinux/syslinux.git;a=blob_plain;f=doc/usbkey.txt;hb=HEAD
 
+Compulab EM-X270 (em-x270)
+==========================
 
-Texas Instruments Beagleboard (beagleboard)
-===========================================
+Fetch the "Linux - kernel and run-time image (Angstrom)" ZIP file from the
+Compulab website. Inside the images directory of this ZIP file is another ZIP
+file called 'LiveDisk.zip'. Extract this over a cleanly formatted vfat USB flash
+drive. Replace the 'em_x270.img' file with the 'updater-em-x270.ext2' file.
 
-The Beagleboard is an ARM Cortex-A8 development board with USB, DVI-D, S-Video,
-2D/3D accelerated graphics, audio, serial, JTAG, and SD/MMC. The xM adds a
-faster CPU, more RAM, an ethernet port, more USB ports, microSD, and removes
-the NAND flash. The beagleboard MACHINE is tested on the following platforms:
+Insert this USB disk into the supplied adapter and connect this to the
+board. Whilst holding down the the suspend button press the reset button. The
+board will now boot off the USB key and into a version of Angstrom. On the
+desktop is an icon labelled "Updater". Run this program to launch the updater
+that will flash the Poky kernel and rootfs to the board.
 
-  o Beagleboard C4
-  o Beagleboard xM rev A & B
 
-The Beagleboard C4 has NAND, while the xM does not. For the sake of simplicity,
-these instructions assume you have erased the NAND on the C4 so its boot
-behavior matches that of the xM. To do this, issue the following commands from
-the u-boot prompt (note that the unlock may be unecessary depending on the
-version of u-boot installed on your board and only one of the erase commands
-will succeed):
+FreeScale iMX31ADS (mx31ads)
+===========================
 
-    # nand unlock
-    # nand erase
-    # nand erase.chip
+The correct serial port is the top-most female connector to the right of the
+ethernet socket.
 
-To further tailor these instructions for your board, please refer to the
-documentation at http://www.beagleboard.org.
+For uploading data to RedBoot we are going to use tftp. In this example we
+assume that the tftpserver is on 192.168.9.1 and the board is on192.168.9.2. 
 
-From a Linux system with access to the image files perform the following steps
-as root, replacing mmcblk0* with the SD card device on your machine (such as sdc
-if used via a usb card reader):
+To set the IP address, run:
 
-  1. Partition and format an SD card:
-     # fdisk -lu /dev/mmcblk0
-     
-     Disk /dev/mmcblk0: 3951 MB, 3951034368 bytes
-     255 heads, 63 sectors/track, 480 cylinders, total 7716864 sectors
-     Units = sectors of 1 * 512 = 512 bytes
-     
-             Device Boot      Start         End      Blocks  Id System
-     /dev/mmcblk0p1   *          63      144584       72261   c Win95 FAT32 (LBA)
-     /dev/mmcblk0p2          144585      465884      160650  83 Linux
+  ip_address -l 192.168.9.2/24 -h 192.168.9.1
 
-     # mkfs.vfat -F 16 -n "boot" /dev/mmcblk0p1
-     # mke2fs -j -L "root" /dev/mmcblk0p2
+To download a kernel called "zimage" from the TFTP server, run:
 
-  The following assumes the SD card partition 1 and 2 are mounted at
-  /media/boot and /media/root respectively. Removing the card and reinserting
-  it will do just that on most modern Linux desktop environments.
-  
-  The files referenced below are made available after the build in
-  build/tmp/deploy/images.
+  load -r -b 0x100000 zimage
 
-  2. Install the boot loaders
-     # cp MLO-beagleboard /media/boot/MLO
-     # cp u-boot-beagleboard.bin /media/boot/u-boot.bin
+To write the kernel to flash run:
 
-  3. Install the root filesystem
-     # tar x -C /media/root -f core-image-$IMAGE_TYPE-beagleboard.tar.bz2
-     # tar x -C /media/root -f modules-$KERNEL_VERSION-beagleboard.tgz
+  fis create kernel
 
-  4. Install the kernel uImage
-     # cp uImage-beagleboard.bin /media/boot/uImage
+To download a rootfs jffs2 image "rootfs" from the TFTP server, run:
 
-  5. Prepare a u-boot script to simplify the boot process
-     The Beagleboard can be made to boot at this point from the u-boot command
-     shell. To automate this process, generate a user.scr script as follows.
+  load -r -b 0x100000 rootfs
 
-     Install uboot-mkimage (from uboot-mkimage on Ubuntu or uboot-tools on Fedora).
+To write the root filesystem  to flash run:
 
-     Prepare a script config:
+  fis create root
 
-     # (cat << EOF
-     setenv bootcmd 'mmc init; fatload mmc 0:1 0x80300000 uImage; bootm 0x80300000'
-     setenv bootargs 'console=tty0 console=ttyO2,115200n8 root=/dev/mmcblk0p2 rootwait rootfstype=ext3 ro'
-     boot
-     EOF
-     ) > serial-boot.cmd
-     # mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n "Core Minimal" -d ./serial-boot.cmd ./boot.scr
-     # cp boot.scr /media/boot
+To load and boot a kernel and rootfs from flash:
 
-   6. Unmount the SD partitions, insert the SD card into the Beagleboard, and
-      boot the Beagleboard
+  fis load kernel
+  exec -b 0x100000 -l 0x200000 -c "noinitrd console=ttymxc0,115200 root=/dev/mtdblock2 rootfstype=jffs2 init=linuxrc ip=none"
 
-Note: As of the 2.6.37 linux-yocto kernel recipe, the Beagleboard uses the
-      OMAP_SERIAL device (ttyO2). If you are using an older kernel, such as the
-      2.6.34 linux-yocto-stable, be sure to replace ttyO2 with ttyS2 above. You
-      should also override the machine SERIAL_CONSOLE in your local.conf in
-      order to setup the getty on the serial line:
+To load and boot a kernel from a TFTP server with the rootfs over NFS:
 
-      SERIAL_CONSOLE_beagleboard = "115200 ttyS2"
+  load -r -b 0x100000 zimage
+  exec -b 0x100000 -l 0x200000 -c "noinitrd console=ttymxc0,115200 root=/dev/nfs nfsroot=192.168.9.1:/mnt/nfsmx31 rw ip=192.168.9.2::192.168.9.1:255.255.255.0"
 
+The instructions above are for using the (default) NOR flash on the board,
+there is also 128M of NAND flash. It is possible to install Poky to the NAND 
+flash which gives more space for the rootfs and instructions for using this are
+given below. To switch to the NAND flash:
 
-Freescale MPC8315E-RDB (mpc8315e-rdb)
-=====================================
+  factive NAND
 
-The MPC8315 PowerPC reference platform (MPC8315E-RDB) is aimed at hardware and
-software development of network attached storage (NAS) and digital media server
-applications. The MPC8315E-RDB features the PowerQUICC II Pro processor, which
-includes a built-in security accelerator.
+This will then restart RedBoot using the NAND rather than the NOR. If you
+have not used the NAND before then it is unlikely that there will be a
+partition table yet. You can get the list of partitions with 'fis list'.
 
-(Note: you may find it easier to order MPC8315E-RDBA; this appears to be the
-same board in an enclosure with accessories. In any case it is fully
-compatible with the instructions given here.)
+If this shows no partitions then you can create them with:
 
-Setup instructions
-------------------
+  fis init
 
-You will need the following:
-* NFS root setup on your workstation
-* TFTP server installed on your workstation
-* Null modem cable connected from your workstation to the first serial port
-  on the board
-* Ethernet connected to the first ethernet port on the board
+The output of 'fis list' should now show:
 
---- Preparation ---
+Name              FLASH addr  Mem addr    Length      Entry point
+RedBoot           0xE0000000  0xE0000000  0x00040000  0x00000000
+FIS directory     0xE7FF4000  0xE7FF4000  0x00003000  0x00000000
+RedBoot config    0xE7FF7000  0xE7FF7000  0x00001000  0x00000000
 
-Note: if you have altered your board's ethernet MAC address(es) from the
-defaults, or you need to do so because you want multiple boards on the same
-network, then you will need to change the values in the dts file (patch
-linux/arch/powerpc/boot/dts/mpc8315erdb.dts within the kernel source). If
-you have left them at the factory default then you shouldn't need to do
-anything here.
+Partitions for the kernel and rootfs need to be created:
 
---- Booting from NFS root ---
+fis create -l 0x1A0000 -e 0x00100000 kernel
+fis create -l 0x5000000 -e 0x00100000 root
 
-Load the kernel and dtb (device tree blob), and boot the system as follows:
+You may now use the instructions above for flashing. However it is important
+to note that the erase block size for the NAND is different to the NOR so the
+JFFS erase size will need to be changed to 0x4000. Stardard images are built
+for NOR and you will need to build custom images for NAND.
 
- 1. Get the kernel (uImage-mpc8315e-rdb.bin) and dtb (uImage-mpc8315e-rdb.dtb)
-    files from the Poky build tmp/deploy directory, and make them available on
-    your TFTP server.
+You will also need to update the kernel command line to use the correct root
+filesystem. This should be '/dev/mtdblock7' if you adhere to the partitioning
+scheme shown above. If this fails then you can doublecheck against the output
+from the kernel when it evaluates the available mtd partitions.
 
- 2. Connect the board's first serial port to your workstation and then start up
-    your favourite serial terminal so that you will be able to interact with
-    the serial console. If you don't have a favourite, picocom is suggested:
 
-  $ picocom /dev/ttyUSB0 -b 115200
+Marvell PXA3xx Zylonite (zylonite)
+==================================
 
- 3. Power up or reset the board and press a key on the terminal when prompted
-    to get to the U-Boot command line
+These instructions assume the Zylonite is connected to a machine running a TFTP 
+server at address 192.168.123.5 and that a serial link (38400 8N1) is available 
+to access the blob bootloader. The kernel is on the TFTP server as 
+"zylonite-kernel" and the root filesystem jffs2 file is "zylonite-rootfs" and 
+the images are to be saved in NAND flash.
 
- 4. Set up the environment in U-Boot:
+The following commands setup blob:
 
- => setenv ipaddr <board ip>
- => setenv serverip <tftp server ip>
- => setenv bootargs root=/dev/nfs rw nfsroot=<nfsroot ip>:<rootfs path> ip=<board ip>:<server ip>:<gateway ip>:255.255.255.0:mpc8315e:eth0:off console=ttyS0,115200
+  blob> setip client 192.168.123.4
+  blob> setip server 192.168.123.5
 
- 5. Download the kernel and dtb, and boot:
+To flash the kernel:
 
- => tftp 800000 uImage-mpc8315e-rdb.bin
- => tftp 780000 uImage-mpc8315e-rdb.dtb
- => bootm 800000 - 780000
+  blob> tftp zylonite-kernel
+  blob> nandwrite -j 0x80800000 0x60000 0x200000
 
+To flash the rootfs:
 
-Ubiquiti Networks RouterStation Pro (routerstationpro)
-======================================================
+  blob> tftp zylonite-rootfs
+  blob> nanderase -j 0x260000 0x5000000
+  blob> nandwrite -j 0x80800000 0x260000 <length>
 
-The RouterStation Pro is an Atheros AR7161 MIPS-based board. Geared towards
-networking applications, it has all of the usual features as well as three
-type IIIA mini-PCI slots and an on-board 3-port 10/100/1000 Ethernet switch,
-in addition to the 10/100/1000 Ethernet WAN port which supports
-Power-over-Ethernet.
+(where <length> is the rootfs size which will be printed by the tftp step)
 
-Setup instructions
-------------------
+To boot the board:
 
-You will need the following:
-* A serial cable - female to female (or female to male + gender changer)
-  NOTE: cable must be straight through, *not* a null modem cable.
-* USB flash drive or hard disk that is able to be powered from the
-  board's USB port.
-* tftp server installed on your workstation
+  blob> nkernel
+  blob> boot
 
-NOTE: in the following instructions it is assumed that /dev/sdb corresponds
-to the USB disk when it is plugged into your workstation. If this is not the
-case in your setup then please be careful to substitute the correct device
-name in all commands where appropriate.
 
---- Preparation ---
+Logic iMX31 Lite Kit (mx31litekit)
+===============================
 
-1) Build an image (e.g. core-image-minimal) using "routerstationpro" as the
-MACHINE
+The easiest method to boot this board is to take an MMC/SD card and format 
+the first partition as ext2, then extract the poky image onto this as root. 
+Assuming the board is network connected, a TFTP server is available at 
+192.168.1.33 and a serial terminal is available (115200 8N1), the following 
+commands will boot a kernel called "mx31kern" from the TFTP server:
 
-2) Partition the USB drive so that primary partition 1 is type Linux (83).
-Minimum size depends on your root image size - core-image-minimal probably
-only needs 8-16MB, other images will need more.
+  losh> ifconfig sm0 192.168.1.203 255.255.255.0 192.168.1.33
+  losh> load raw 0x80100000 0x200000 /tftp/192.168.1.33:mx31kern
+  losh> exec 0x80100000 - 
 
-  # fdisk /dev/sdb
-  Command (m for help): p
 
-  Disk /dev/sdb: 4011 MB, 4011491328 bytes
-  124 heads, 62 sectors/track, 1019 cylinders, total 7834944 sectors
-  Units = sectors of 1 * 512 = 512 bytes
-  Sector size (logical/physical): 512 bytes / 512 bytes
-  I/O size (minimum/optimal): 512 bytes / 512 bytes
-  Disk identifier: 0x0009e87d
+Phytec phyCORE-iMX31 (mx31phy)
+==============================
 
-     Device Boot      Start         End      Blocks   Id  System
-  /dev/sdb1              62     1952751      976345   83  Linux
+Support for this board is currently being developed. Experimental jffs2 
+images and a suitable kernel are available and are known to work with the
+board.
 
-3) Format partition 1 on the USB as ext3
 
-  # mke2fs -j /dev/sdb1
+                             Consumer Devices
+                             ================
 
-4) Mount partition 1 and then extract the contents of
-tmp/deploy/images/core-image-XXXX.tar.bz2 into it (preserving permissions).
+FIC Neo1973 GTA01 smartphone (fic-gta01)
+========================================
 
-  # mount /dev/sdb1 /media/sdb1
-  # cd /media/sdb1
-  # tar -xvjpf tmp/deploy/images/core-image-XXXX.tar.bz2
+To install Poky on a GTA01 smartphone you will need "dfu-util" tool
+which you can build with "bitbake dfu-util-native" command.
 
-5) Unmount the USB drive and then plug it into the board's USB port
+Flashing requires these steps:
 
-6) Connect the board's serial port to your workstation and then start up
-your favourite serial terminal so that you will be able to interact with
-the serial console. If you don't have a favourite, picocom is suggested:
+  1. Power down the device.
+  2. Connect the device to the host machine via USB.
+  3. Hold AUX key and press Power key. There should be a bootmenu
+     on screen.
+  4. Run "dfu-util -l" to check if the phone is visible on the USB bus. 
+     The output should look like this:
 
-  $ picocom /dev/ttyUSB0 -b 115200
+     dfu-util - (C) 2007 by OpenMoko Inc.
+     This program is Free Software and has ABSOLUTELY NO WARRANTY
 
-7) Connect the network into eth0 (the one that is NOT the 3 port switch). If
-you are using power-over-ethernet then the board will power up at this point.
+     Found Runtime: [0x1457:0x5119] devnum=19, cfg=0, intf=2, alt=0, name="USB Device Firmware Upgrade"
 
-8) Start up the board, watch the serial console. Hit Ctrl+C to abort the
-autostart if the board is configured that way (it is by default). The
-bootloader's fconfig command can be used to disable autostart and configure
-the IP settings if you need to change them (default IP is 192.168.1.20).
+  5. Flash the kernel with "dfu-util -a kernel -D uImage-2.6.21.6-moko11-r2-fic-gta01.bin"
+  6. Flash rootfs with "dfu-util -a rootfs -D <image>", where <image> is the
+     jffs2 image file to use as the root filesystem 
+     (e.g. ./tmp/deploy/images/poky-image-sato-fic-gta01.jffs2) 
 
-9) Make the kernel (tmp/deploy/images/vmlinux-routerstationpro.bin) available
-on the tftp server.
 
-10) If you are going to write the kernel to flash (optional - see "Booting a
-kernel directly" below for the alternative), remove the current kernel and
-rootfs flash partitions. You can list the partitions using the following
-bootloader command:
+HTC Universal (htcuniversal)
+============================
 
-  RedBoot> fis list
+Note: HTC Universal support is highly experimental.
 
-You can delete the existing kernel and rootfs with these commands:
+On the HTC Universal, entirely replacing the Windows installation is not 
+supported, instead Poky is booted from an MMC/SD card from Windows. Once Poky
+has booted, Windows is no longer in memory or active but when power is removed,
+the user will be returned to windows and will need to return to Linux from 
+there.
 
-  RedBoot> fis delete kernel
-  RedBoot> fis delete rootfs
+Once an MMC/SD card is available it is suggested its split into two partitions,
+one for a program called HaRET which lets you boot Linux from within Windows
+and the second for the rootfs. The HaRET partition should be the first partition
+on the card and be vfat formatted. It doesn't need to be large, just enough for
+HaRET and a kernel (say 5MB max). The rootfs should be ext2 and is usually the 
+second partition. The first partition should be vfat so Windows recognises it 
+as if it doesn't, it has been known to reformat cards.
 
---- Booting a kernel directly ---
+On the first partition you need three files:
 
-1) Load the kernel using the following bootloader command:
+  * a HaRET binary (version 0.5.1 works well and a working version 
+    should be part of the last Poky release)
+  * a kernel renamed to "zImage"
+  * a default.txt which contains:
 
-  RedBoot> load -m tftp -h <ip of tftp server> vmlinux-routerstationpro.bin
+set kernel "zImage"
+set mtype "855"
+set cmdline "root=/dev/mmcblk0p2 rw console=ttyS0,115200n8 console=tty0 rootdelay=5 fbcon=rotate:1"
+boot2
 
-You should see a message on it being successfully loaded.
+On the second parition the root file system is extracted as root. A different
+partition layout or other kernel options can be changed in the default.txt file.
 
-2) Execute the kernel:
+When inserted into the device, Windows should see the card and let you browse
+its contents using File Explorer. Running the HaRET binary will present a dialog
+box (maybe after messages warning about running unsigned binaries) where you 
+select OK and you should then see Poky boot. Kernel messages can be seen by 
+adding psplash=false to the kernel commandline.
 
-  RedBoot> exec -c "console=ttyS0,115200 root=/dev/sda1 rw rootdelay=2 board=UBNT-RSPRO"
 
-Note that specifying the command line with -c is important as linux-yocto does
-not provide a default command line.
+Nokia 770/N800/N810 Internet Tablets (nokia770 and nokia800)
+============================================================
 
---- Writing a kernel to flash ---
+Note: Nokia tablet support is highly experimental.
 
-1) Go to your tftp server and gzip the kernel you want in flash. It should
-halve the size.
+The Nokia internet tablet devices are OMAP based tablet formfactor devices 
+with large screens (800x480), wifi and touchscreen.
 
-2) Load the kernel using the following bootloader command:
+To flash images to these devices you need the "flasher" utility which can be 
+downloaded from the http://tablets-dev.nokia.com/d3.php?f=flasher-3.0. This 
+utility needs to be run as root and the usb filesystem needs to be mounted 
+although most distributions will have done this for you. Once you have this 
+follow these steps:
 
-  RedBoot> load -r -b 0x80600000 -m tftp -h <ip of tftp server> vmlinux-routerstationpro.bin.gz
+  1. Power down the device.
+  2. Connect the device to the host machine via USB 
+     (connecting power to the device doesn't hurt either).
+  3. Run "flasher -i"
+  4. Power on the device.
+  5. The program should give an indication it's found 
+     a tablet device. If not, recheck the cables, make sure you're 
+     root and usbfs/usbdevfs is mounted.
+  6. Run "flasher -r <image> -k <kernel> -f", where <image> is the
+     jffs2 image file to use as the root filesystem 
+     (e.g. ./tmp/deploy/images/poky-image-sato-nokia800.jffs2) 
+     and <kernel> is the kernel to use 
+     (e.g. ./tmp/deploy/images/zImage-nokia800.bin).
+  7. Run "flasher -R" to reboot the device.
+  8. The device should boot into Poky.
 
-This should output something similar to the following:
+The nokia800 images and kernel will run on both the N800 and N810.
 
-  Raw file loaded 0x80600000-0x8087c537, assumed entry at 0x80600000
 
-Calculate the length by subtracting the first number from the second number
-and then rounding the result up to the nearest 0x1000.
+Sharp Zaurus SL-C7x0 series (c7x0)
+==================================
 
-3) Using the length calculated above, create a flash partition for the kernel:
+The Sharp Zaurus c7x0 series (SL-C700, SL-C750, SL-C760, SL-C860, SL-7500)
+are PXA25x based handheld PDAs with VGA screens. To install Poky images on
+these devices follow these steps:
 
-  RedBoot> fis create -b 0x80600000 -l 0x240000 kernel
+  1. Obtain an SD/MMC or CF card with a vfat or ext2 filesystem.
+  2. Copy a jffs2 image file (e.g. poky-image-sato-c7x0.jffs2) onto the
+     card as "initrd.bin":
 
-(change 0x240000 to your rounded length -- change "kernel" to whatever
-you want to name your kernel)
+     $ cp ./tmp/deploy/images/poky-image-sato-c7x0.jffs2 /path/to/my-cf-card/initrd.bin
 
---- Booting a kernel from flash ---
+  3. Copy an Linux kernel file (zImage-c7x0.bin) onto the card as 
+     "zImage.bin":
 
-To boot the flashed kernel perform the following steps.
+     $ cp ./tmp/deploy/images/zImage-c7x0.bin /path/to/my-cf-card/zImage.bin
 
-1) At the bootloader prompt, load the kernel:
+  4. Copy an updater script (updater.sh.c7x0) onto the card
+     as "updater.sh":
 
-  RedBoot> fis load -d -e kernel
+     $ cp ./tmp/deploy/images/updater.sh.c7x0 /path/to/my-cf-card/updater.sh
 
-(Change the name "kernel" above if you chose something different earlier)
+  5. Power down the Zaurus.
+  6. Hold "OK" key and power on the device. An update menu should appear 
+     (in Japanese).
+  7. Choose "Update" (item 4).
+  8. The next screen will ask for the source, choose the appropriate 
+     card (CF or SD).
+  9. Make sure AC power is connected.
+  10. The next screen asks for confirmation, choose "Yes" (the left button).
+  11. The update process will start, flash the files on the card onto 
+      the device and the device will then reboot into Poky.
 
-(-e means 'elf', -d 'decompress')
 
-2) Execute the kernel using the exec command as above.
+Sharp Zaurus SL-C1000 (akita)
+=============================
 
---- Automating the boot process ---
+The Sharp Zaurus SL-C1000 is a PXA270 based device otherwise similar to the 
+c7x0. To install Poky images on this device follow the instructions for 
+the c7x0 but replace "c7x0" with "akita" where appropriate.
+
+
+Sharp Zaurus SL-C3x00 series (spitz)
+====================================
+
+The Sharp Zaurus SL-C3x00 devices are PXA270 based devices similar
+to akita but with an internal microdrive. The installation procedure 
+assumes a standard microdrive based device where the root (first) 
+partition has been enlarged to fit the image (at least 100MB, 
+400MB for the SDK).
+
+The procedure is the same as for the c7x0 and akita models with the 
+following differences:
+
+ 1. Instead of a jffs2 image you need to copy a compressed tarball of the 
+    root fileystem (e.g. poky-image-sato-spitz.tar.gz) onto the
+    card as "hdimage1.tgz":
+
+    $ cp ./tmp/deploy/images/poky-image-sato-spitz.tar.gz /path/to/my-cf-card/hdimage1.tgz
+
+ 2. You additionally need to copy a special tar utility  (gnu-tar) onto 
+    the card as "gnu-tar":
+
+    $ cp ./tmp/deploy/images/gnu-tar /path/to/my-cf-card/gnu-tar
 
-After writing the kernel to flash and testing the load and exec commands
-manually, you can automate the boot process with a boot script.
 
-1) RedBoot> fconfig
-   (Answer the questions not specified here as they pertain to your environment)
-2) Run script at boot: true
-  Boot script: 
-  .. fis load -d -e kernel
-  .. exec
-  Enter script, terminate with empty line
-  >> fis load -d -e kernel
-  >> exec -c "console=ttyS0,115200 root=/dev/sda1 rw rootdelay=2 board=UBNT-RSPRO"
-  >> 
-3) Answer the remaining questions and write the changes to flash:
-  Update RedBoot non-volatile configuration - continue (y/n)? y
-  ... Erase from 0xbfff0000-0xc0000000: .
-  ... Program from 0x87ff0000-0x88000000 at 0xbfff0000: .
-4) Power cycle the board.
 

bitbake/AUTHORS

@@ -1,7 +1,7 @@
 Tim Ansell <mithro@mithis.net>
 Phil Blundell <pb@handhelds.org>
 Seb Frankengul <seb@frankengul.org>
-Holger Freyther <holger@moiji-mobile.com>
+Holger Freyther <zecke@handhelds.org>
 Marcin Juszkiewicz <marcin@juszkiewicz.com.pl>
 Chris Larson <kergoth@handhelds.org>
 Ulrich Luckas <luckas@musoft.de>

bitbake/bin/bitbake

@@ -23,27 +23,28 @@
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import os
-import sys, logging
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)),
+import sys
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])),
                                 'lib'))
 
 import optparse
 import warnings
 from traceback import format_exception
-try:
-    import bb
-except RuntimeError as exc:
-    sys.exit(str(exc))
-from bb import event
+import bb
 import bb.msg
 from bb import cooker
 from bb import ui
 from bb import server
+from bb.server import none
+#from bb.server import xmlrpc
 
-__version__ = "1.13.3"
-logger = logging.getLogger("BitBake")
+__version__ = "1.11.0"
 
 
+
+#============================================================================#
+# BBOptions
+#============================================================================#
 class BBConfiguration(object):
     """
     Manages build options and configurations for one run
@@ -55,44 +56,34 @@ class BBConfiguration(object):
         self.pkgs_to_build = []
 
 
-def get_ui(config):
-    if config.ui:
-        interface = config.ui
-    else:
-        interface = 'knotty'
+def print_exception(exc, value, tb):
+    """Send exception information through bb.msg"""
+    bb.fatal("".join(format_exception(exc, value, tb, limit=8)))
 
-    try:
-        # Dynamically load the UI based on the ui name. Although we
-        # suggest a fixed set this allows you to have flexibility in which
-        # ones are available.
-        module = __import__("bb.ui", fromlist = [interface])
-        return getattr(module, interface).main
-    except AttributeError:
-        sys.exit("FATAL: Invalid user interface '%s' specified.\n"
-                 "Valid interfaces: depexp, goggle, ncurses, knotty [default]." % interface)
+sys.excepthook = print_exception
 
 
-# Display bitbake/OE warnings via the BitBake.Warnings logger, ignoring others"""
-warnlog = logging.getLogger("BitBake.Warnings")
 _warnings_showwarning = warnings.showwarning
 def _showwarning(message, category, filename, lineno, file=None, line=None):
+    """Display python warning messages using bb.msg"""
     if file is not None:
         if _warnings_showwarning is not None:
             _warnings_showwarning(message, category, filename, lineno, file, line)
     else:
         s = warnings.formatwarning(message, category, filename, lineno)
-        warnlog.warn(s)
+        s = s.split("\n")[0]
+        bb.msg.warn(None, s)
 
 warnings.showwarning = _showwarning
-warnings.filterwarnings("ignore")
-warnings.filterwarnings("default", module="(<string>$|(oe|bb)\.)")
-warnings.filterwarnings("ignore", category=PendingDeprecationWarning)
-warnings.filterwarnings("ignore", category=ImportWarning)
-warnings.filterwarnings("ignore", category=DeprecationWarning, module="<string>$")
-warnings.filterwarnings("ignore", message="With-statements now directly support multiple context managers")
+warnings.simplefilter("ignore", DeprecationWarning)
 
+#============================================================================#
+# main
+#============================================================================#
 
 def main():
+    return_value = 1
+
     parser = optparse.OptionParser(
         version = "BitBake Build Tool Core version %s, %%prog version %s" % (bb.__version__, __version__),
         usage = """%prog [options] [package ...]
@@ -102,7 +93,7 @@ It expects that BBFILES is defined, which is a space separated list of files to
 be executed.  BBFILES does support wildcards.
 Default BBFILES are the .bb files in the current directory.""")
 
-    parser.add_option("-b", "--buildfile", help = "execute the task against this .bb file, rather than a package from BBFILES. Does not handle any dependencies.",
+    parser.add_option("-b", "--buildfile", help = "execute the task against this .bb file, rather than a package from BBFILES.",
                action = "store", dest = "buildfile", default = None)
 
     parser.add_option("-k", "--continue", help = "continue as much as possible after an error. While the target that failed, and those that depend on it, cannot be remade, the other dependencies of these targets can be processed all the same.",
@@ -118,10 +109,7 @@ Default BBFILES are the .bb files in the current directory.""")
                action = "store", dest = "cmd")
 
     parser.add_option("-r", "--read", help = "read the specified file before bitbake.conf",
-               action = "append", dest = "prefile", default = [])
-
-    parser.add_option("-R", "--postread", help = "read the specified file after bitbake.conf",
-                      action = "append", dest = "postfile", default = [])
+               action = "append", dest = "file", default = [])
 
     parser.add_option("-v", "--verbose", help = "output more chit-chat to the terminal",
                action = "store_true", dest = "verbose", default = False)
@@ -138,6 +126,9 @@ Default BBFILES are the .bb files in the current directory.""")
     parser.add_option("-p", "--parse-only", help = "quit after parsing the BB files (developers only)",
                action = "store_true", dest = "parse_only", default = False)
 
+    parser.add_option("-d", "--disable-psyco", help = "disable using the psyco just-in-time compiler (not recommended)",
+               action = "store_true", dest = "disable_psyco", default = False)
+
     parser.add_option("-s", "--show-versions", help = "show current and preferred versions of all packages",
                action = "store_true", dest = "show_versions", default = False)
 
@@ -159,9 +150,6 @@ Default BBFILES are the .bb files in the current directory.""")
     parser.add_option("-u", "--ui", help = "userinterface to use",
                action = "store", dest = "ui")
 
-    parser.add_option("-t", "--servertype", help = "Choose which server to use, none, process or xmlrpc",
-               action = "store", dest = "servertype")
-
     parser.add_option("", "--revisions-changed", help = "Set the exit code depending on whether upstream floating revisions have changed or not",
                action = "store_true", dest = "revisions_changed", default = False)
 
@@ -169,76 +157,61 @@ Default BBFILES are the .bb files in the current directory.""")
 
     configuration = BBConfiguration(options)
     configuration.pkgs_to_build.extend(args[1:])
+    configuration.initial_path = os.environ['PATH']
 
-    ui_main = get_ui(configuration)
-
-    # Server type can be xmlrpc, process or none currently, if nothing is specified,
-    # the default server is process
-    if configuration.servertype:
-        server_type = configuration.servertype
-    else:
-        server_type = 'process'
-
-    try:
-        module = __import__("bb.server", fromlist = [server_type])
-        server = getattr(module, server_type)
-    except AttributeError:
-        sys.exit("FATAL: Invalid server type '%s' specified.\n"
-                 "Valid interfaces: xmlrpc, process [default], none." % servertype)
+    #server = bb.server.xmlrpc
+    server = bb.server.none
 
     # Save a logfile for cooker into the current working directory. When the
     # server is daemonized this logfile will be truncated.
     cooker_logfile = os.path.join(os.getcwd(), "cooker.log")
 
-    bb.msg.init_msgconfig(configuration.verbose, configuration.debug,
+    bb.utils.init_logger(bb.msg, configuration.verbose, configuration.debug,
                          configuration.debug_domains)
 
-    # Ensure logging messages get sent to the UI as events
-    handler = bb.event.LogHandler()
-    logger.addHandler(handler)
-
-    # Before we start modifying the environment we should take a pristine
-    # copy for possible later use
-    initialenv = os.environ.copy()
     # Clear away any spurious environment variables. But don't wipe the
     # environment totally. This is necessary to ensure the correct operation
     # of the UIs (e.g. for DISPLAY, etc.)
     bb.utils.clean_environment()
 
-    server = server.BitBakeServer()
+    cooker = bb.cooker.BBCooker(configuration, server)
 
-    server.initServer()
-    idle = server.getServerIdleCB()
-
-    cooker = bb.cooker.BBCooker(configuration, idle, initialenv)
     cooker.parseCommandLine()
 
-    server.addcooker(cooker)
-    server.saveConnectionDetails()
-    server.detach(cooker_logfile)
+    serverinfo = server.BitbakeServerInfo(cooker.server)
 
-    # Should no longer need to ever reference cooker
+    server.BitBakeServerFork(serverinfo, cooker.serve, cooker_logfile)
     del cooker
 
-    logger.removeHandler(handler)
-
     # Setup a connection to the server (cooker)
-    server_connection = server.establishConnection()
+    serverConnection = server.BitBakeServerConnection(serverinfo)
+
+    # Launch the UI
+    if configuration.ui:
+        ui = configuration.ui
+    else:
+        ui = "knotty"
 
     try:
-        return server.launchUI(ui_main, server_connection.connection, server_connection.events)
+        # Dynamically load the UI based on the ui name. Although we
+        # suggest a fixed set this allows you to have flexibility in which
+        # ones are available.
+        uimodule = __import__("bb.ui", fromlist = [ui])
+        ui_init = getattr(uimodule, ui).init
+    except AttributeError:
+        print("FATAL: Invalid user interface '%s' specified. " % ui)
+        print("Valid interfaces are 'ncurses', 'depexp' or the default, 'knotty'.")
+    else:
+        try:
+            return_value = ui_init(serverConnection.connection, serverConnection.events)
+        except Exception as e:
+            print("FATAL: Unable to start to '%s' UI: %s" % (ui, e))
+            raise
     finally:
-        bb.event.ui_queue = []
-        server_connection.terminate()
+        serverConnection.terminate()
 
-    return 1
+    return return_value
 
 if __name__ == "__main__":
-    try:
-        ret = main()
-    except Exception:
-        ret = 1
-        import traceback
-        traceback.print_exc(5)
+    ret = main()
     sys.exit(ret)
-

bitbake/bin/bitbake-layers

@@ -1,299 +0,0 @@
-#!/usr/bin/env python
-
-# This script has subcommands which operate against your bitbake layers, either
-# displaying useful information, or acting against them.
-# See the help output for details on available commands.
-
-import cmd
-import logging
-import os
-import sys
-
-bindir = os.path.dirname(__file__)
-topdir = os.path.dirname(bindir)
-sys.path[0:0] = [os.path.join(topdir, 'lib')]
-
-import bb.cache
-import bb.cooker
-import bb.providers
-import bb.utils
-from bb.cooker import state
-
-
-logger = logging.getLogger('BitBake')
-
-
-def main(args):
-    # Set up logging
-    console = logging.StreamHandler(sys.stdout)
-    format = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
-    bb.msg.addDefaultlogFilter(console)
-    console.setFormatter(format)
-    logger.addHandler(console)
-
-    initialenv = os.environ.copy()
-    bb.utils.clean_environment()
-
-    cmds = Commands(initialenv)
-    if args:
-        cmds.onecmd(' '.join(args))
-    else:
-        cmds.do_help('')
-    return cmds.returncode
-
-
-class Commands(cmd.Cmd):
-    def __init__(self, initialenv):
-        cmd.Cmd.__init__(self)
-        self.returncode = 0
-        self.config = Config(parse_only=True)
-        self.cooker = bb.cooker.BBCooker(self.config,
-                                         self.register_idle_function,
-                                         initialenv)
-        self.config_data = self.cooker.configuration.data
-        bb.providers.logger.setLevel(logging.ERROR)
-        self.cooker_data = None
-
-    def register_idle_function(self, function, data):
-        pass
-
-    def prepare_cooker(self):
-        sys.stderr.write("Parsing recipes..")
-        logger.setLevel(logging.WARNING)
-
-        try:
-            while self.cooker.state in (state.initial, state.parsing):
-                self.cooker.updateCache()
-        except KeyboardInterrupt:
-            self.cooker.shutdown()
-            self.cooker.updateCache()
-            sys.exit(2)
-
-        logger.setLevel(logging.INFO)
-        sys.stderr.write("done.\n")
-
-        self.cooker_data = self.cooker.status
-        self.cooker_data.appends = self.cooker.appendlist
-
-    def check_prepare_cooker(self):
-        if not self.cooker_data:
-            self.prepare_cooker()
-
-    def default(self, line):
-        """Handle unrecognised commands"""
-        sys.stderr.write("Unrecognised command or option\n")
-        self.do_help('')
-
-    def do_help(self, topic):
-        """display general help or help on a specified command"""
-        if topic:
-            sys.stdout.write('%s: ' % topic)
-            cmd.Cmd.do_help(self,topic)
-        else:
-            sys.stdout.write("usage: bitbake-layers <command> [arguments]\n\n")
-            sys.stdout.write("Available commands:\n")
-            procnames = self.get_names()
-            for procname in procnames:
-                if procname[:3] == 'do_':
-                    sys.stdout.write("  %s\n" % procname[3:])
-                    doc = getattr(self, procname).__doc__
-                    if doc:
-                        sys.stdout.write("    %s\n" % doc.splitlines()[0])
-
-    def do_show_layers(self, args):
-        """show current configured layers"""
-        self.check_prepare_cooker()
-        logger.plain('')
-        logger.plain("%s  %s  %s" % ("layer".ljust(20), "path".ljust(40), "priority"))
-        logger.plain('=' * 74)
-        layerdirs = str(self.config_data.getVar('BBLAYERS', True)).split()
-        for layerdir in layerdirs:
-            layername = '?'
-            layerpri = 0
-            for layer, _, regex, pri in self.cooker.status.bbfile_config_priorities:
-                if regex.match(os.path.join(layerdir, 'test')):
-                    layername = layer
-                    layerpri = pri
-                    break
-
-            logger.plain("%s  %s  %d" % (layername.ljust(20), layerdir.ljust(40), layerpri))
-
-    def do_show_overlayed(self, args):
-        """list overlayed recipes (where there is a recipe in another layer that has a higher layer priority)
-
-usage: show_overlayed
-
-Highest priority recipes are listed with the recipes they overlay as subitems.
-"""
-        self.check_prepare_cooker()
-        if self.cooker.overlayed:
-            logger.plain('Overlayed recipes:')
-            for f in self.cooker.overlayed.iterkeys():
-                logger.plain('%s' % f)
-                for of in self.cooker.overlayed[f]:
-                    logger.plain('  %s' % of)
-        else:
-            logger.plain('No overlayed recipes found')
-
-    def do_flatten(self, args):
-        """flattens layer configuration into a separate output directory.
-
-usage: flatten <outputdir>
-
-Takes the current layer configuration and builds a "flattened" directory
-containing the contents of all layers, with any overlayed recipes removed
-and bbappends appended to the corresponding recipes. Note that some manual
-cleanup may still be necessary afterwards, in particular:
-
-* where non-recipe files (such as patches) are overwritten (the flatten
-  command will show a warning for these)
-* where anything beyond the normal layer setup has been added to
-  layer.conf (only the lowest priority layer's layer.conf is used)
-* overridden/appended items from bbappends will need to be tidied up
-"""
-        arglist = args.split()
-        if len(arglist) != 1:
-            logger.error('Please specify an output directory')
-            self.do_help('flatten')
-            return
-
-        if os.path.exists(arglist[0]) and os.listdir(arglist[0]):
-            logger.error('Directory %s exists and is non-empty, please clear it out first' % arglist[0])
-            return
-
-        self.check_prepare_cooker()
-        layers = (self.config_data.getVar('BBLAYERS', True) or "").split()
-        for layer in layers:
-            overlayed = []
-            for f in self.cooker.overlayed.iterkeys():
-                for of in self.cooker.overlayed[f]:
-                    if of.startswith(layer):
-                        overlayed.append(of)
-
-            logger.plain('Copying files from %s...' % layer )
-            for root, dirs, files in os.walk(layer):
-                for f1 in files:
-                    f1full = os.sep.join([root, f1])
-                    if f1full in overlayed:
-                        logger.plain('  Skipping overlayed file %s' % f1full )
-                    else:
-                        ext = os.path.splitext(f1)[1]
-                        if ext != '.bbappend':
-                            fdest = f1full[len(layer):]
-                            fdest = os.path.normpath(os.sep.join([arglist[0],fdest]))
-                            bb.utils.mkdirhier(os.path.dirname(fdest))
-                            if os.path.exists(fdest):
-                                if f1 == 'layer.conf' and root.endswith('/conf'):
-                                    logger.plain('  Skipping layer config file %s' % f1full )
-                                    continue
-                                else:
-                                    logger.warn('Overwriting file %s', fdest)
-                            bb.utils.copyfile(f1full, fdest)
-                            if ext == '.bb':
-                                if f1 in self.cooker_data.appends:
-                                    appends = self.cooker_data.appends[f1]
-                                    if appends:
-                                        logger.plain('  Applying appends to %s' % fdest )
-                                        for appendname in appends:
-                                            self.apply_append(appendname, fdest)
-
-    def get_append_layer(self, appendname):
-        for layer, _, regex, _ in self.cooker.status.bbfile_config_priorities:
-            if regex.match(appendname):
-                return layer
-        return "?"
-
-    def apply_append(self, appendname, recipename):
-        appendfile = open(appendname, 'r')
-        recipefile = open(recipename, 'a')
-        recipefile.write('\n')
-        recipefile.write('##### bbappended from %s #####\n' % self.get_append_layer(appendname))
-        recipefile.writelines(appendfile.readlines())
-
-    def do_show_appends(self, args):
-        """list bbappend files and recipe files they apply to
-
-usage: show_appends
-
-Recipes are listed with the bbappends that apply to them as subitems.
-"""
-        self.check_prepare_cooker()
-        if not self.cooker_data.appends:
-            logger.plain('No append files found')
-            return
-
-        logger.plain('State of append files:')
-
-        pnlist = list(self.cooker_data.pkg_pn.keys())
-        pnlist.sort()
-        for pn in pnlist:
-            self.show_appends_for_pn(pn)
-
-        self.show_appends_for_skipped()
-
-    def show_appends_for_pn(self, pn):
-        filenames = self.cooker_data.pkg_pn[pn]
-
-        best = bb.providers.findBestProvider(pn,
-                                             self.cooker.configuration.data,
-                                             self.cooker_data,
-                                             self.cooker_data.pkg_pn)
-        best_filename = os.path.basename(best[3])
-
-        self.show_appends_output(filenames, best_filename)
-
-    def show_appends_for_skipped(self):
-        filenames = [os.path.basename(f)
-                    for f in self.cooker.skiplist.iterkeys()]
-        self.show_appends_output(filenames, None, " (skipped)")
-
-    def show_appends_output(self, filenames, best_filename, name_suffix = ''):
-        appended, missing = self.get_appends_for_files(filenames)
-        if appended:
-            for basename, appends in appended:
-                logger.plain('%s%s:', basename, name_suffix)
-                for append in appends:
-                    logger.plain('  %s', append)
-
-            if best_filename:
-                if best_filename in missing:
-                    logger.warn('%s: missing append for preferred version',
-                                best_filename)
-                    self.returncode |= 1
-
-
-    def get_appends_for_files(self, filenames):
-        appended, notappended = [], []
-        for filename in filenames:
-            _, cls = bb.cache.Cache.virtualfn2realfn(filename)
-            if cls:
-                continue
-
-            basename = os.path.basename(filename)
-            appends = self.cooker_data.appends.get(basename)
-            if appends:
-                appended.append((basename, list(appends)))
-            else:
-                notappended.append(basename)
-        return appended, notappended
-
-
-class Config(object):
-    def __init__(self, **options):
-        self.pkgs_to_build = []
-        self.debug_domains = []
-        self.extra_assume_provided = []
-        self.prefile = []
-        self.postfile = []
-        self.debug = 0
-        self.__dict__.update(options)
-
-    def __getattr__(self, attribute):
-        try:
-            return super(Config, self).__getattribute__(attribute)
-        except AttributeError:
-            return None
-
-
-if __name__ == '__main__':
-    sys.exit(main(sys.argv[1:]) or 0)

bitbake/bin/bitbake-prserv

@@ -1,53 +0,0 @@
-#!/usr/bin/env python
-import os
-import sys,logging
-import optparse
-
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)),'lib'))
-
-import prserv
-import prserv.serv
-
-__version__="1.0.0"
-
-PRHOST_DEFAULT=''
-PRPORT_DEFAULT=8585
-
-def main():
-    parser = optparse.OptionParser(
-        version="Bitbake PR Service Core version %s, %%prog version %s" % (prserv.__version__, __version__),
-        usage = "%prog [options]")
-
-    parser.add_option("-f", "--file", help="database filename(default prserv.db)", action="store",
-                      dest="dbfile", type="string", default="prserv.db")
-    parser.add_option("-l", "--log", help="log filename(default prserv.log)", action="store",
-                      dest="logfile", type="string", default="prserv.log")
-    parser.add_option("--loglevel", help="logging level, i.e. CRITICAL, ERROR, WARNING, INFO, DEBUG",
-                      action = "store", type="string", dest="loglevel", default = "WARNING")
-    parser.add_option("--start", help="start daemon",
-                      action="store_true", dest="start", default="True")
-    parser.add_option("--stop", help="stop daemon",
-                      action="store_false", dest="start")
-    parser.add_option("--host", help="ip address to bind", action="store",
-                      dest="host", type="string", default=PRHOST_DEFAULT)
-    parser.add_option("--port", help="port number(default 8585)", action="store",
-                      dest="port", type="int", default=PRPORT_DEFAULT)
-
-    options, args = parser.parse_args(sys.argv)
-
-    prserv.init_logger(os.path.abspath(options.logfile),options.loglevel)
-
-    if options.start:
-        prserv.serv.start_daemon(options)
-    else:
-        prserv.serv.stop_daemon()
-
-if __name__ == "__main__":
-    try:
-        ret = main()
-    except Exception:
-        ret = 1
-        import traceback
-        traceback.print_exc(5)
-    sys.exit(ret)
-

bitbake/bin/bitbake-runtask

@@ -4,10 +4,6 @@ import os
 import sys
 import warnings
 sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
-from bb import fetch2
-import logging
-
-logger = logging.getLogger("BitBake")
 
 try:
     import cPickle as pickle
@@ -20,20 +16,13 @@ class BBConfiguration(object):
     Manages build options and configurations for one run
     """
 
-    def __init__(self, **options):
-        self.data = {}
-        self.file = []
-        self.cmd = None
-        self.dump_signatures = True
-        self.prefile = []
-        self.postfile = []
-        self.parse_only = True
-
-    def __getattr__(self, attribute):
-        try:
-            return super(BBConfiguration, self).__getattribute__(attribute)
-        except AttributeError:
-            return None
+    def __init__(self, debug, debug_domains):
+        setattr(self, "data", {})
+        setattr(self, "file", [])
+        setattr(self, "cmd", None)
+        setattr(self, "dump_signatures", True)
+        setattr(self, "debug", debug)
+        setattr(self, "debug_domains", debug_domains)
 
 _warnings_showwarning = warnings.showwarning
 def _showwarning(message, category, filename, lineno, file=None, line=None):
@@ -50,70 +39,79 @@ warnings.showwarning = _showwarning
 warnings.simplefilter("ignore", DeprecationWarning)
 
 import bb.event
+
+# Need to map our I/O correctly. stdout is a pipe to the server expecting 
+# events. We save this and then map stdout to stderr.
+
+eventfd = os.dup(sys.stdout.fileno())
+bb.event.worker_pipe = os.fdopen(eventfd, 'w', 0)
+
+# map stdout to stderr
+os.dup2(sys.stderr.fileno(), sys.stdout.fileno())
+
+# Replace those fds with our own
+#logout = data.expand("${TMPDIR}/log/stdout.%s" % os.getpid(), self.cfgData, True)
+#mkdirhier(os.path.dirname(logout))
+#newso = open("/tmp/stdout.%s" % os.getpid(), 'w')
+#os.dup2(newso.fileno(), sys.stdout.fileno())
+#os.dup2(newso.fileno(), sys.stderr.fileno())
+
+# Don't read from stdin from the parent
+si = file("/dev/null", 'r')
+os.dup2(si.fileno( ), sys.stdin.fileno( ))
+
+# We don't want to see signals to our parent, e.g. Ctrl+C
+os.setpgrp()
+
+# Save out the PID so that the event can include it the
+# events
+bb.event.worker_pid = os.getpid()
+bb.event.useStdout = False
+
+hashfile = sys.argv[1]
+buildfile = sys.argv[2]
+taskname = sys.argv[3]
+
 import bb.cooker
 
-buildfile = sys.argv[1]
-taskname = sys.argv[2]
-if len(sys.argv) >= 4:
-    dryrun = sys.argv[3]    
-else:
-    dryrun = False
-if len(sys.argv) >= 5:
-    hashfile = sys.argv[4]
-    p = pickle.Unpickler(file(hashfile, "rb"))
-    hashdata = p.load()
-else:
-    hashdata = None
+p = pickle.Unpickler(file(hashfile, "rb"))
+hashdata = p.load()
 
-handler = bb.event.LogHandler()
-logger.addHandler(handler)
+debug = hashdata["msg-debug"]
+debug_domains = hashdata["msg-debug-domains"]
+verbose = hashdata["verbose"]
 
-#An example to make debug log messages show up
-#bb.msg.init_msgconfig(True, 3, [])
+bb.utils.init_logger(bb.msg, verbose, debug, debug_domains)
 
-console = logging.StreamHandler(sys.stdout)
-format = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
-bb.msg.addDefaultlogFilter(console)
-console.setFormatter(format)
+cooker = bb.cooker.BBCooker(BBConfiguration(debug, debug_domains), None)
+cooker.parseConfiguration()
 
-def worker_fire(event, d):
-    if isinstance(event, logging.LogRecord):
-        console.handle(event)
-bb.event.worker_fire = worker_fire
-bb.event.worker_pid = os.getpid()
+cooker.bb_cache = bb.cache.init(cooker)
+cooker.status = bb.cache.CacheData()
 
-initialenv = os.environ.copy()
-config = BBConfiguration()
-
-def register_idle_function(self, function, data):
-    pass
-
-cooker = bb.cooker.BBCooker(config, register_idle_function, initialenv)
-config_data = cooker.configuration.data
-cooker.status = config_data
-cooker.handleCollections(bb.data.getVar("BBFILE_COLLECTIONS", config_data, 1))
-
-fn, cls = bb.cache.Cache.virtualfn2realfn(buildfile)
+(fn, cls) = cooker.bb_cache.virtualfn2realfn(buildfile)
 buildfile = cooker.matchFile(fn)
-fn = bb.cache.Cache.realfn2virtual(buildfile, cls)
+fn = cooker.bb_cache.realfn2virtual(buildfile, cls)
 
 cooker.buildSetVars()
 
 # Load data into the cache for fn and parse the loaded cache data
-the_data = bb.cache.Cache.loadDataFull(fn, cooker.get_file_appends(fn), cooker.configuration.data)
+the_data = cooker.bb_cache.loadDataFull(fn, cooker.get_file_appends(fn), cooker.configuration.data)
+cooker.bb_cache.setData(fn, buildfile, the_data)
+cooker.bb_cache.handle_data(fn, cooker.status)
 
 if taskname.endswith("_setscene"):
     the_data.setVarFlag(taskname, "quieterrors", "1")
 
-if hashdata:
-    bb.parse.siggen.set_taskdata(hashdata["hashes"], hashdata["deps"])
-    for h in hashdata["hashes"]:
-        bb.data.setVar("BBHASH_%s" % h, hashdata["hashes"][h], the_data)
-    for h in hashdata["deps"]:
-        bb.data.setVar("BBHASHDEPS_%s" % h, hashdata["deps"][h], the_data)
+bb.parse.siggen.set_taskdata(hashdata["hashes"], hashdata["deps"])
+
+for h in hashdata["hashes"]:
+    bb.data.setVar("BBHASH_%s" % h, hashdata["hashes"][h], the_data)
+for h in hashdata["deps"]:
+    bb.data.setVar("BBHASHDEPS_%s" % h, hashdata["deps"][h], the_data)
 
 ret = 0
-if dryrun != "True":
+if sys.argv[4] != "True":
     ret = bb.build.exec_task(fn, taskname, the_data)
 sys.exit(ret)
 

bitbake/bin/bitdoc

@@ -20,7 +20,7 @@
 import optparse, os, sys
 
 # bitbake
-sys.path.append(os.path.join(os.path.dirname(os.path.dirname(__file__), 'lib'))
+sys.path.append(os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
 import bb
 import bb.parse
 from   string import split, join
@@ -430,8 +430,9 @@ Create a set of html pages (documentation) for a bitbake.conf....
                        action = "store_true", dest = "verbose", default = False )
 
     options, args = parser.parse_args( sys.argv )
- 
-    bb.msg.init_msgconfig(options.verbose, options.debug)
+
+    if options.debug:
+        bb.msg.set_debug_level(options.debug)
 
     return options.config, options.output
 
@@ -496,7 +497,7 @@ def main():
         doc.insert_doc_item(doc_ins)
 
     # let us create the HTML now
-    bb.utils.mkdirhier(output_dir)
+    bb.mkdirhier(output_dir)
     os.chdir(output_dir)
 
     # Let us create the sites now. We do it in the following order

bitbake/contrib/vim/ftdetect/bitbake.vim

@@ -1,24 +1,4 @@
-" Vim filetype detection file
-" Language:     BitBake
-" Author:       Ricardo Salveti <rsalveti@rsalveti.net>
-" Copyright:    Copyright (C) 2008  Ricardo Salveti <rsalveti@rsalveti.net>
-" Licence:      You may redistribute this under the same terms as Vim itself
-"
-" This sets up the syntax highlighting for BitBake files, like .bb, .bbclass and .inc
-
-if &compatible || version < 600
-    finish
-endif
-
-" .bb and .bbclass
-au BufNewFile,BufRead *.b{b,bclass}	set filetype=bitbake
-
-" .inc
-au BufNewFile,BufRead *.inc 		set filetype=bitbake
-
-" .conf
-au BufNewFile,BufRead *.conf
-    \ if (match(expand("%:p:h"), "conf") > 0) |
-    \     set filetype=bitbake |
-    \ endif
-
+au BufNewFile,BufRead *.bb       setfiletype bitbake
+au BufNewFile,BufRead *.bbclass  setfiletype bitbake
+au BufNewFile,BufRead *.inc      setfiletype bitbake
+" au BufNewFile,BufRead *.conf     setfiletype bitbake

bitbake/contrib/vim/ftplugin/bitbake.vim

@@ -1 +0,0 @@
-set sts=4 sw=4 et

bitbake/contrib/vim/plugin/newbb.vim

@@ -1,85 +0,0 @@
-" Vim plugin file
-" Purpose:	Create a template for new bb files
-" Author:	Ricardo Salveti <rsalveti@gmail.com>
-" Copyright:	Copyright (C) 2008 Ricardo Salveti <rsalveti@gmail.com>
-"
-" This file is licensed under the MIT license, see COPYING.MIT in
-" this source distribution for the terms.
-"
-" Based on the gentoo-syntax package
-"
-" Will try to use git to find the user name and email
-
-if &compatible || v:version < 600
-    finish
-endif
-
-fun! <SID>GetUserName()
-    let l:user_name = system("git-config --get user.name")
-    if v:shell_error
-        return "Unknow User"
-    else
-        return substitute(l:user_name, "\n", "", "")
-endfun
-
-fun! <SID>GetUserEmail()
-    let l:user_email = system("git-config --get user.email")
-    if v:shell_error
-        return "unknow@user.org"
-    else
-        return substitute(l:user_email, "\n", "", "")
-endfun
-
-fun! BBHeader()
-    let l:current_year = strftime("%Y")
-    let l:user_name = <SID>GetUserName()
-    let l:user_email = <SID>GetUserEmail()
-    0 put ='# Copyright (C) ' . l:current_year .
-                \ ' ' . l:user_name . ' <' . l:user_email . '>'
-    put ='# Released under the MIT license (see COPYING.MIT for the terms)'
-    $
-endfun
-
-fun! NewBBTemplate()
-    let l:paste = &paste
-    set nopaste
-    
-    " Get the header
-    call BBHeader()
-
-    " New the bb template
-    put ='DESCRIPTION = \"\"'
-    put ='HOMEPAGE = \"\"'
-    put ='LICENSE = \"\"' 
-    put ='SECTION = \"\"'
-    put ='DEPENDS = \"\"'
-    put ='PR = \"r0\"'
-    put =''
-    put ='SRC_URI = \"\"'
-
-    " Go to the first place to edit
-    0
-    /^DESCRIPTION =/
-    exec "normal 2f\""
-
-    if paste == 1
-        set paste
-    endif
-endfun
-
-if !exists("g:bb_create_on_empty")
-    let g:bb_create_on_empty = 1
-endif
-
-" disable in case of vimdiff
-if v:progname =~ "vimdiff"
-    let g:bb_create_on_empty = 0
-endif
-
-augroup NewBB
-    au BufNewFile *.bb
-                \ if g:bb_create_on_empty |
-                \    call NewBBTemplate() |
-                \ endif
-augroup END
-

bitbake/contrib/vim/syntax/bitbake.vim

@@ -1,123 +1,127 @@
 " Vim syntax file
-" Language:     BitBake bb/bbclasses/inc
-" Author:       Chris Larson <kergoth@handhelds.org>
-"               Ricardo Salveti <rsalveti@rsalveti.net>
-" Copyright:    Copyright (C) 2004  Chris Larson <kergoth@handhelds.org>
-"               Copyright (C) 2008  Ricardo Salveti <rsalveti@rsalveti.net>
 "
+" Copyright (C) 2004  Chris Larson <kergoth@handhelds.org>
 " This file is licensed under the MIT license, see COPYING.MIT in
 " this source distribution for the terms.
 "
-" Syntax highlighting for bb, bbclasses and inc files.
-"
-" It's an entirely new type, just has specific syntax in shell and python code
+" Language:	BitBake
+" Maintainer:	Chris Larson <kergoth@handhelds.org>
+" Filenames:	*.bb, *.bbclass
 
-if &compatible || v:version < 600
-    finish
-endif
-if exists("b:current_syntax")
-    finish
+if version < 600
+  syntax clear
+elseif exists("b:current_syntax")
+  finish
 endif
 
+syn case match
+
+" Catch incorrect syntax (only matches if nothing else does)
+"
+syn match bbUnmatched		"."
+
+
 syn include @python syntax/python.vim
 if exists("b:current_syntax")
   unlet b:current_syntax
 endif
 
-" BitBake syntax
 
-" Matching case
-syn case match
+" Other
 
-" Indicates the error when nothing is matched
-syn match bbUnmatched           "."
+syn match bbComment		"^#.*$" display contains=bbTodo
+syn keyword bbTodo		TODO FIXME XXX contained
+syn match bbDelimiter		"[(){}=]" contained
+syn match bbQuote		/['"]/ contained
+syn match bbArrayBrackets	"[\[\]]" contained
 
-" Comments
-syn cluster bbCommentGroup      contains=bbTodo,@Spell
-syn keyword bbTodo              COMBAK FIXME TODO XXX contained
-syn match bbComment             "#.*$" contains=@bbCommentGroup
-
-" String helpers
-syn match bbQuote               +['"]+ contained 
-syn match bbDelimiter           "[(){}=]" contained
-syn match bbArrayBrackets       "[\[\]]" contained
 
 " BitBake strings
-syn match bbContinue            "\\$"
-syn region bbString             matchgroup=bbQuote start=+"+ skip=+\\$+ excludenl end=+"+ contained keepend contains=bbTodo,bbContinue,bbVarDeref,bbVarPyValue,@Spell
-syn region bbString             matchgroup=bbQuote start=+'+ skip=+\\$+ excludenl end=+'+ contained keepend contains=bbTodo,bbContinue,bbVarDeref,bbVarPyValue,@Spell
 
-" Vars definition
-syn match bbExport            "^export" nextgroup=bbIdentifier skipwhite
-syn keyword bbExportFlag        export contained nextgroup=bbIdentifier skipwhite
-syn match bbIdentifier          "[a-zA-Z0-9\-_\.\/\+]\+" display contained
-syn match bbVarDeref            "${[a-zA-Z0-9\-_\.\/\+]\+}" contained
-syn match bbVarEq               "\(:=\|+=\|=+\|\.=\|=\.\|?=\|=\)" contained nextgroup=bbVarValue
-syn match bbVarDef              "^\(export\s*\)\?\([a-zA-Z0-9\-_\.\/\+]\+\(_[${}a-zA-Z0-9\-_\.\/\+]\+\)\?\)\s*\(:=\|+=\|=+\|\.=\|=\.\|?=\|=\)\@=" contains=bbExportFlag,bbIdentifier,bbVarDeref nextgroup=bbVarEq
-syn match bbVarValue            ".*$" contained contains=bbString,bbVarDeref,bbVarPyValue
-syn region bbVarPyValue         start=+${@+ skip=+\\$+ excludenl end=+}+ contained contains=@python
+syn match bbContinue		"\\$"
+syn region bbString		matchgroup=bbQuote start=/"/ skip=/\\$/ excludenl end=/"/ contained keepend contains=bbTodo,bbContinue,bbVarInlinePy,bbVarDeref
+syn region bbString		matchgroup=bbQuote start=/'/ skip=/\\$/ excludenl end=/'/ contained keepend contains=bbTodo,bbContinue,bbVarInlinePy,bbVarDeref
 
-" Vars metadata flags
-syn match bbVarFlagDef          "^\([a-zA-Z0-9\-_\.]\+\)\(\[[a-zA-Z0-9\-_\.]\+\]\)\@=" contains=bbIdentifier nextgroup=bbVarFlagFlag
-syn region bbVarFlagFlag        matchgroup=bbArrayBrackets start="\[" end="\]\s*\(=\)\@=" keepend excludenl contained contains=bbIdentifier nextgroup=bbVarEq
+" BitBake variable metadata
 
-" Includes and requires
-syn keyword bbInclude           inherit include require contained 
-syn match bbIncludeRest         ".*$" contained contains=bbString,bbVarDeref
-syn match bbIncludeLine         "^\(inherit\|include\|require\)\s\+" contains=bbInclude nextgroup=bbIncludeRest
+syn match bbVarBraces "[\${}]"
+syn region bbVarDeref matchgroup=bbVarBraces start="${" end="}" contained
+" syn region bbVarDeref start="${" end="}" contained
+" syn region bbVarInlinePy start="${@" end="}" contained contains=@python
+syn region bbVarInlinePy matchgroup=bbVarBraces start="${@" end="}" contained contains=@python
 
-" Add taks and similar
-syn keyword bbStatement         addtask addhandler after before EXPORT_FUNCTIONS contained
-syn match bbStatementRest       ".*$" skipwhite contained contains=bbStatement
-syn match bbStatementLine       "^\(addtask\|addhandler\|after\|before\|EXPORT_FUNCTIONS\)\s\+" contains=bbStatement nextgroup=bbStatementRest
+syn keyword bbExportFlag	export contained nextgroup=bbIdentifier skipwhite
+" syn match bbVarDeref	"${[a-zA-Z0-9\-_\.]\+}" contained
+syn match bbVarDef		"^\(export\s*\)\?\([a-zA-Z0-9\-_\.]\+\(_[${}a-zA/-Z0-9\-_\.]\+\)\?\)\s*\(:=\|+=\|=+\|\.=\|=\.\|?=\|=\)\@=" contains=bbExportFlag,bbIdentifier,bbVarDeref nextgroup=bbVarEq
 
-" OE Important Functions
-syn keyword bbOEFunctions       do_fetch do_unpack do_patch do_configure do_compile do_stage do_install do_package contained
+syn match bbIdentifier		"[a-zA-Z0-9\-_\./]\+" display contained
+"syn keyword bbVarEq	= display contained nextgroup=bbVarValue
+syn match bbVarEq		"\(:=\|+=\|=+\|\.=\|=\.\|?=\|=\)" contained nextgroup=bbVarValue
+syn match bbVarValue		".*$" contained contains=bbString
+
+" BitBake variable metadata flags
+syn match bbVarFlagDef		"^\([a-zA-Z0-9\-_\.]\+\)\(\[[a-zA-Z0-9\-_\.]\+\]\)\@=" contains=bbIdentifier nextgroup=bbVarFlagFlag
+syn region bbVarFlagFlag	matchgroup=bbArrayBrackets start="\[" end="\]\s*\(=\)\@=" keepend excludenl contained contains=bbIdentifier nextgroup=bbVarEq
+"syn match bbVarFlagFlag		"\[\([a-zA-Z0-9\-_\.]\+\)\]\s*\(=\)\@=" contains=bbIdentifier nextgroup=bbVarEq
+
+
+" Functions!
+syn match bbFunction	"\h\w*" display contained
+
+
+" BitBake python metadata
+
+syn keyword bbPythonFlag	python contained nextgroup=bbFunction
+syn match bbPythonFuncDef	"^\(python\s\+\)\(\w\+\)\?\(\s*()\s*\)\({\)\@=" contains=bbPythonFlag,bbFunction,bbDelimiter nextgroup=bbPythonFuncRegion skipwhite
+syn region bbPythonFuncRegion	matchgroup=bbDelimiter start="{\s*$" end="^}\s*$" keepend contained contains=@python
+"hi def link bbPythonFuncRegion	Comment
 
-" Generic Functions
-syn match bbFunction            "\h[0-9A-Za-z_-]*" display contained contains=bbOEFunctions
 
 " BitBake shell metadata
 syn include @shell syntax/sh.vim
 if exists("b:current_syntax")
   unlet b:current_syntax
 endif
-syn keyword bbShFakeRootFlag    fakeroot contained
-syn match bbShFuncDef           "^\(fakeroot\s*\)\?\([0-9A-Za-z_-]\+\)\(python\)\@<!\(\s*()\s*\)\({\)\@=" contains=bbShFakeRootFlag,bbFunction,bbDelimiter nextgroup=bbShFuncRegion skipwhite
-syn region bbShFuncRegion       matchgroup=bbDelimiter start="{\s*$" end="^}\s*$" keepend contained contains=@shell
 
-" BitBake python metadata
-syn keyword bbPyFlag            python contained
-syn match bbPyFuncDef           "^\(python\s\+\)\([0-9A-Za-z_-]\+\)\?\(\s*()\s*\)\({\)\@=" contains=bbPyFlag,bbFunction,bbDelimiter nextgroup=bbPyFuncRegion skipwhite 
-syn region bbPyFuncRegion       matchgroup=bbDelimiter start="{\s*$" end="^}\s*$" keepend contained contains=@python
+syn keyword bbFakerootFlag	fakeroot contained nextgroup=bbFunction
+syn match bbShellFuncDef	"^\(fakeroot\s*\)\?\(\w\+\)\(python\)\@<!\(\s*()\s*\)\({\)\@=" contains=bbFakerootFlag,bbFunction,bbDelimiter nextgroup=bbShellFuncRegion skipwhite
+syn region bbShellFuncRegion	matchgroup=bbDelimiter start="{\s*$" end="^}\s*$" keepend contained contains=@shell
+"hi def link bbShellFuncRegion	Comment
+
 
 " BitBake 'def'd python functions
-syn keyword bbPyDef             def contained
-syn region bbPyDefRegion        start='^\(def\s\+\)\([0-9A-Za-z_-]\+\)\(\s*(.*)\s*\):\s*$' end='^\(\s\|$\)\@!' contains=@python
+syn keyword bbDef	def	contained
+syn region bbDefRegion		start='^def\s\+\w\+\s*([^)]*)\s*:\s*$' end='^\(\s\|$\)\@!' contains=@python
 
-" Highlighting Definitions
-hi def link bbUnmatched         Error
-hi def link bbInclude           Include
-hi def link bbTodo              Todo
-hi def link bbComment           Comment
-hi def link bbQuote             String
-hi def link bbString            String
-hi def link bbDelimiter         Keyword
-hi def link bbArrayBrackets     Statement
-hi def link bbContinue          Special
-hi def link bbExport            Type
-hi def link bbExportFlag        Type
-hi def link bbIdentifier	    Identifier
-hi def link bbVarDeref          PreProc
-hi def link bbVarDef            Identifier
-hi def link bbVarValue          String
-hi def link bbShFakeRootFlag    Type
-hi def link bbFunction          Function
-hi def link bbPyFlag            Type
-hi def link bbPyDef             Statement
-hi def link bbStatement         Statement
-hi def link bbStatementRest     Identifier
-hi def link bbOEFunctions       Special
-hi def link bbVarPyValue        PreProc
+
+" BitBake statements
+syn keyword bbStatement		include inherit require addtask addhandler EXPORT_FUNCTIONS display contained
+syn match bbStatementLine	"^\(include\|inherit\|require\|addtask\|addhandler\|EXPORT_FUNCTIONS\)\s\+" contains=bbStatement nextgroup=bbStatementRest
+syn match bbStatementRest		".*$" contained contains=bbString,bbVarDeref
+
+" Highlight
+"
+hi def link bbArrayBrackets	Statement
+hi def link bbUnmatched		Error
+hi def link bbContinue		Special
+hi def link bbDef		Statement
+hi def link bbPythonFlag	Type
+hi def link bbExportFlag	Type
+hi def link bbFakerootFlag	Type
+hi def link bbStatement		Statement
+hi def link bbString		String
+hi def link bbTodo		Todo
+hi def link bbComment		Comment
+hi def link bbOperator		Operator
+hi def link bbError		Error
+hi def link bbFunction		Function
+hi def link bbDelimiter		Delimiter
+hi def link bbIdentifier	Identifier
+hi def link bbVarEq		Operator
+hi def link bbQuote		String
+hi def link bbVarValue		String
+" hi def link bbVarInlinePy	PreProc
+hi def link bbVarDeref		PreProc
+hi def link bbVarBraces		PreProc
 
 let b:current_syntax = "bb"

bitbake/doc/bitbake.1

@@ -85,6 +85,9 @@ don't execute, just go through the motions
 .B \-p, \-\-parse-only
 quit after parsing the BB files (developers only)
 .TP
+.B \-d, \-\-disable-psyco
+disable using the psyco just-in-time compiler (not recommended)
+.TP
 .B \-s, \-\-show-versions
 show current and preferred versions of all packages
 .TP

bitbake/doc/manual/Makefile

@@ -45,7 +45,7 @@ endif
 	$(call command,xsltproc --stringparam base.dir $@/ $(if $(htmlcssfile),--stringparam html.stylesheet $(htmlcssfile)) $(htmlxsl) $(manual),XSLTPROC $@ $(manual))
 
 $(xmltotypes): $(manual)
-	$(call command,xmlto --with-dblatex --extensions -o $(topdir)/$@ $@ $(manual),XMLTO $@ $(manual))
+	$(call command,xmlto --extensions -o $(topdir)/$@ $@ $(manual),XMLTO $@ $(manual))
 
 clean:
 	rm -rf $(cleanfiles)

bitbake/doc/manual/usermanual.xml

@@ -12,10 +12,9 @@
             <corpauthor>BitBake Team</corpauthor>
         </authorgroup>
         <copyright>
-            <year>2004, 2005, 2006, 2011</year>
+            <year>2004, 2005, 2006</year>
             <holder>Chris Larson</holder>
             <holder>Phil Blundell</holder>
-            <holder>Richard Purdie</holder>
         </copyright>
         <legalnotice>
             <para>This work is licensed under the Creative Commons Attribution License. To view a copy of this license, visit <ulink url="http://creativecommons.org/licenses/by/2.5/">http://creativecommons.org/licenses/by/2.5/</ulink> or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.</para>
@@ -27,10 +26,10 @@
             <title>Overview</title>
             <para>BitBake is, at its simplest, a tool for executing
 tasks and managing metadata.  As such, its similarities to GNU make and other
-build tools are readily apparent.  It was inspired by Portage, the package management system used by the Gentoo Linux distribution.  BitBake is the basis of the <ulink url="http://www.openembedded.org/">OpenEmbedded</ulink> project, which is being used to build and maintain a number of embedded Linux distributions/projects such as Angstrom and the Yocto project.</para>
+build tools are readily apparent.  It was inspired by Portage, the package management system used by the Gentoo Linux distribution.  BitBake is the basis of the <ulink url="http://www.openembedded.org/">OpenEmbedded</ulink> project, which is being used to build and maintain a number of embedded Linux distributions, including OpenZaurus and Familiar.</para>
         </section>
         <section>
-            <title>Background and goals</title>
+            <title>Background and Goals</title>
             <para>Prior to BitBake, no other build tool adequately met
 the needs of an aspiring embedded Linux distribution.  All of the
 buildsystems used by traditional desktop Linux distributions lacked
@@ -38,14 +37,14 @@ important functionality, and none of the ad-hoc
 <emphasis>buildroot</emphasis> systems, prevalent in the
 embedded space, were scalable or maintainable.</para>
 
-      <para>Some important original goals for BitBake were:
+      <para>Some important goals for BitBake were:
             <itemizedlist>
                 <listitem><para>Handle crosscompilation.</para></listitem>
                 <listitem><para>Handle interpackage dependencies (build time on target architecture, build time on native architecture, and runtime).</para></listitem>
                 <listitem><para>Support running any number of tasks within a given package, including, but not limited to, fetching upstream sources, unpacking them, patching them, configuring them, et cetera.</para></listitem>
-                <listitem><para>Must be Linux distribution agnostic (both build and target).</para></listitem>
+                <listitem><para>Must be linux distribution agnostic (both build and target).</para></listitem>
                 <listitem><para>Must be architecture agnostic</para></listitem>
-                <listitem><para>Must support multiple build and target operating systems (including Cygwin, the BSDs, etc).</para></listitem>
+                <listitem><para>Must support multiple build and target operating systems (including cygwin, the BSDs, etc).</para></listitem>
                 <listitem><para>Must be able to be self contained, rather than tightly integrated into the build machine's root filesystem.</para></listitem>
                 <listitem><para>There must be a way to handle conditional metadata (on target architecture, operating system, distribution, machine).</para></listitem>
                 <listitem><para>It must be easy for the person using the tools to supply their own local metadata and packages to operate against.</para></listitem>
@@ -54,18 +53,10 @@ between multiple projects using BitBake for their
 builds.</para></listitem>
 		<listitem><para>Should provide an inheritance mechanism to
 share common metadata between many packages.</para></listitem>
+                <listitem><para>Et cetera...</para></listitem>
             </itemizedlist>
         </para>
-        <para>Over time it has become apparent that some further requirements were necessary:
-            <itemizedlist>
-                <listitem><para>Handle variants of a base recipe (native, sdk, multilib).</para></listitem>
-                <listitem><para>Able to split metadata into layers and allow layers to override each other.</para></listitem>
-                <listitem><para>Allow representation of a given set of input variables to a task as a checksum.</para></listitem>
-                <listitem><para>based on that checksum, allow acceleration of builds with prebuilt components.</para></listitem>
-            </itemizedlist>
-        </para>
-
-        <para>BitBake satisfies all the original requirements and many more with extensions being made to the basic functionality to reflect the additionl requirements.  Flexibility and power have always been the priorities.  It is highly extensible, supporting embedded Python code and execution of any arbitrary tasks.</para>
+        <para>BitBake satisfies all these and many more.  Flexibility and power have always been the priorities.  It is highly extensible, supporting embedded Python code and execution of any arbitrary tasks.</para>
         </section>
     </chapter>
     <chapter>
@@ -100,13 +91,13 @@ share common metadata between many packages.</para></listitem>
             <section>
                 <title>Setting a default value (?=)</title>
                 <para><screen><varname>A</varname> ?= "aval"</screen></para>
-                <para>If <varname>A</varname> is set before the above is called, it will retain its previous value. If <varname>A</varname> is unset prior to the above call, <varname>A</varname> will be set to <literal>aval</literal>.  Note that this assignment is immediate, so if there are multiple ?= assignments to a single variable, the first of those will be used.</para>
+                <para>If <varname>A</varname> is set before the above is called, it will retain it's previous value. If <varname>A</varname> is unset prior to the above call, <varname>A</varname> will be set to <literal>aval</literal>.  Note that this assignment is immediate, so if there are multiple ?= assignments to a single variable, the first of those will be used.</para>
             </section>
             <section>
-                <title>Setting a weak default value (??=)</title>
-                <para><screen><varname>A</varname> ??= "somevalue"
-<varname>A</varname> ??= "someothervalue"</screen></para>
-                <para>If <varname>A</varname> is set before the above, it will retain that value.  If <varname>A</varname> is unset prior to the above, <varname>A</varname> will be set to <literal>someothervalue</literal>.  This is a lazy/weak assignment in that the assignment does not occur until the end of the parsing process, so that the last, rather than the first, ??= assignment to a given variable will be used. Any other setting of A using = or ?= will however override the value set with ??=</para>
+                <title>Setting a default value (??=)</title>
+                <para><screen><varname>A</varname> ??= "somevalue"</screen></para>
+                <para><screen><varname>A</varname> ??= "someothervalue"</screen></para>
+                <para>If <varname>A</varname> is set before the above, it will retain that value.  If <varname>A</varname> is unset prior to the above, <varname>A</varname> will be set to <literal>someothervalue</literal>.  This is a lazy version of ??=, in that the assignment does not occur until the end of the parsing process, so that the last, rather than the first, ??= assignment to a given variable will be used.</para>
             </section>
             <section>
                 <title>Immediate variable expansion (:=)</title>
@@ -134,7 +125,7 @@ share common metadata between many packages.</para></listitem>
 <varname>B</varname> .= "additionaldata"
 <varname>C</varname> = "cval"
 <varname>C</varname> =. "test"</screen></para>
-                <para>In this example, <varname>B</varname> is now <literal>bvaladditionaldata</literal> and <varname>C</varname> is <literal>testcval</literal>. In contrast to the above appending and prepending operators, no additional space
+                <para>In this example, <varname>B</varname> is now <literal>bvaladditionaldata</literal> and <varname>C</varname> is <literal>testcval</literal>. In contrast to the above Appending and Prepending operators no additional space
 will be introduced.</para>
             </section>
             <section>
@@ -156,12 +147,12 @@ will be introduced.</para>
             </section>
             <section>
                 <title>Inclusion</title>
-                <para>Next, there is the <literal>include</literal> directive, which causes BitBake to parse whatever file you specify, and insert it at that location, which is not unlike <command>make</command>.  However, if the path specified on the <literal>include</literal> line is a relative path, BitBake will locate the first one it can find within <envar>BBPATH</envar>.</para>
+                <para>Next, there is the <literal>include</literal> directive, which causes BitBake to parse in whatever file you specify, and insert it at that location, which is not unlike <command>make</command>.  However, if the path specified on the <literal>include</literal> line is a relative path, BitBake will locate the first one it can find within <envar>BBPATH</envar>.</para>
             </section>
             <section>
-                <title>Requiring inclusion</title>
+                <title>Requiring Inclusion</title>
                 <para>In contrast to the <literal>include</literal> directive, <literal>require</literal> will
-raise an ParseError if the file to be included cannot be found. Otherwise it will behave just like the <literal>
+raise an ParseError if the to be included file can not be found. Otherwise it will behave just like the <literal>
 include</literal> directive.</para>
             </section>
             <section>
@@ -180,10 +171,10 @@ include</literal> directive.</para>
     import time
     print time.strftime('%Y%m%d', time.gmtime())
 }</screen></para>
-                <para>This is the similar to the previous, but flags it as Python so that BitBake knows it is Python code.</para>
+                <para>This is the similar to the previous, but flags it as python so that BitBake knows it is python code.</para>
             </section>
             <section>
-                <title>Defining Python functions into the global Python namespace</title>
+                <title>Defining python functions into the global python namespace</title>
                 <para><emphasis>NOTE:</emphasis> This is only supported in .bb and .bbclass files.</para>
                 <para><screen>def get_depends(bb, d):
     if bb.data.getVar('SOMECONDITION', d, True):
@@ -196,8 +187,8 @@ include</literal> directive.</para>
                 <para>This would result in <varname>DEPENDS</varname> containing <literal>dependencywithcond</literal>.</para>
             </section>
             <section>
-                <title>Variable flags</title>
-                <para>Variables can have associated flags which provide a way of tagging extra information onto a variable. Several flags are used internally by BitBake but they can be used externally too if needed. The standard operations mentioned above also work on flags.</para>
+                <title>Variable Flags</title>
+                <para>Variables can have associated flags which provide a way of tagging extra information onto a variable. Several flags are used internally by bitbake but they can be used externally too if needed. The standard operations mentioned above also work on flags.</para>
                 <para><screen><varname>VARIABLE</varname>[<varname>SOMEFLAG</varname>] = "value"</screen></para>
                 <para>In this example, <varname>VARIABLE</varname> has a flag, <varname>SOMEFLAG</varname> which is set to <literal>value</literal>.</para>
             </section>
@@ -209,32 +200,19 @@ include</literal> directive.</para>
             <section>
                 <title>Tasks</title>
                 <para><emphasis>NOTE:</emphasis> This is only supported in .bb and .bbclass files.</para>
-                <para>In BitBake, each step that needs to be run for a given .bb is known as a task.  There is a command <literal>addtask</literal> to add new tasks (must be a defined Python executable metadata and must start with <quote>do_</quote>) and describe intertask dependencies.</para>
+                <para>In BitBake, each step that needs to be run for a given .bb is known as a task.  There is a command <literal>addtask</literal> to add new tasks (must be a defined python executable metadata and must start with <quote>do_</quote>) and describe intertask dependencies.</para>
                 <para><screen>python do_printdate () {
     import time
     print time.strftime('%Y%m%d', time.gmtime())
 }
 
 addtask printdate before do_build</screen></para>
-                <para>This defines the necessary Python function and adds it as a task which is now a dependency of do_build, the default task.  If anyone executes the do_build task, that will result in do_printdate being run first.</para>
+                <para>This defines the necessary python function and adds it as a task which is now a dependency of do_build (the default task).  If anyone executes the do_build task, that will result in do_printdate being run first.</para>
             </section>
-
-            <section>
-                <title>Task Flags</title>
-                <para>Tasks support a number of flags which control various functionality of the task. These are as follows:</para>
-                <para>'dirs' - directories which should be created before the task runs</para>
-                <para>'cleandirs' - directories which should created before the task runs but should be empty</para>
-                <para>'noexec' - marks the tasks as being empty and no execution required. These are used as dependency placeholders or used when added tasks need to be subsequently disabled.</para>
-                <para>'nostamp' - don't generate a stamp file for a task. This means the task is always rexecuted.</para>
-                <para>'fakeroot' - this task needs to be run in a fakeroot environment, obtained by adding the variables in FAKEROOTENV to the environment.</para>
-                <para>'umask' - the umask to run the task under.</para>
-                <para> For the 'deptask', 'rdeptask', 'recdeptask' and 'recrdeptask' flags please see the dependencies section.</para>
-            </section>
-
             <section>
                 <title>Events</title>
                 <para><emphasis>NOTE:</emphasis> This is only supported in .bb and .bbclass files.</para>
-                <para>BitBake allows installation of event handlers.  Events are triggered at certain points during operation, such as the beginning of operation against a given .bb, the start of a given task, task failure, task success, et cetera.  The intent is to make it easy to do things like email notification on build failure.</para>
+                <para>BitBake allows to install event handlers.  Events are triggered at certain points during operation, such as, the beginning of operation against a given .bb, the start of a given task, task failure, task success, et cetera.  The intent was to make it easy to do things like email notifications on build failure.</para>
                 <para><screen>addhandler myclass_eventhandler
 python myclass_eventhandler() {
     from bb.event import getName
@@ -250,65 +228,20 @@ of the event and the content of the <varname>FILE</varname> variable.</para>
             </section>
             <section>
                 <title>Variants</title>
-                <para>Two BitBake features exist to facilitate the creation of multiple buildable incarnations from a single recipe file.</para>
-                <para>The first is <varname>BBCLASSEXTEND</varname>.  This variable is a space separated list of classes used to "extend" the recipe for each variant.  As an example, setting <screen>BBCLASSEXTEND = "native"</screen> results in a second incarnation of the current recipe being available.  This second incarantion will have the "native" class inherited.</para>
-                <para>The second feature is <varname>BBVERSIONS</varname>.  This variable allows a single recipe to build multiple versions of a project from a single recipe file, and allows you to specify conditional metadata (using the <varname>OVERRIDES</varname> mechanism) for a single version, or an optionally named range of versions:</para>
+                <para>Two Bitbake features exist to facilitate the creation of multiple buildable incarnations from a single recipe file.</para>
+                <para>The first is <varname>BBCLASSEXTEND</varname>.  This variable is a space separated list of classes to utilize to "extend" the recipe for each variant.  As an example, setting <screen>BBCLASSEXTEND = "native"</screen> results in a second incarnation of the current recipe being available.  This second incarantion will have the "native" class inherited.</para>
+                <para>The second feature is <varname>BBVERSIONS</varname>.  This variable allows a single recipe to be able to build multiple versions of a project from a single recipe file, and allows you to specify conditional metadata (using the <varname>OVERRIDES</varname> mechanism) for a single version, or an optionally named range of versions:</para>
                 <para><screen>BBVERSIONS = "1.0 2.0 git"
 SRC_URI_git = "git://someurl/somepath.git"</screen></para>
                 <para><screen>BBVERSIONS = "1.0.[0-6]:1.0.0+ \
               1.0.[7-9]:1.0.7+"
 SRC_URI_append_1.0.7+ = "file://some_patch_which_the_new_versions_need.patch;patch=1"</screen></para>
-                <para>Note that the name of the range will default to the original version of the recipe, so given OE, a recipe file of foo_1.0.0+.bb will default the name of its versions to 1.0.0+.  This is useful, as the range name is not only placed into overrides; it's also made available for the metadata to use in the form of the <varname>BPV</varname> variable, for use in file:// search paths (<varname>FILESPATH</varname>).</para>
+                <para>Note that the name of the range will default to the original version of the recipe, so given OE, a recipe file of foo_1.0.0+.bb will default the name of its versions to 1.0.0+.  This is useful, as the range name is not only placed into overrides, it's also made available for the metadata to use in the form of the <varname>BPV</varname> variable, for use in file:// search paths (<varname>FILESPATH</varname>).</para>
             </section>
         </section>
-
         <section>
-            <title>Variable interaction: Worked Examples</title>
-            <para>Despite the documentation of the different forms of variable definition above, it can be hard to work out what happens when variable operators are combined. This section documents some common questions people have regarding the way variables interact.</para>
-
-            <section>
-                <title>Override and append ordering</title>
-
-                <para>There is often confusion about which order overrides and the various append operators take effect.</para>
-
-                <para><screen><varname>OVERRIDES</varname> = "foo"
-<varname>A_foo_append</varname> = "X"</screen></para>
-                <para>In this case, X is unconditionally appended to the variable <varname>A_foo</varname>. Since foo is an override, A_foo would then replace <varname>A</varname>.</para>
-
-                 <para><screen><varname>OVERRIDES</varname> = "foo"
-<varname>A</varname> = "X"
-<varname>A_append_foo</varname> = "Y"</screen></para>
-                <para>In this case, only when foo is in OVERRIDES, Y is appended to the variable <varname>A</varname> so the value of <varname>A</varname> would become XY (NB: no spaces are appended).</para>
-
-                <para><screen><varname>OVERRIDES</varname> = "foo"
-<varname>A_foo_append</varname> = "X"
-<varname>A_foo_append</varname> += "Y"</screen></para>
-                <para>This behaves as per the first case above, but the value of <varname>A</varname> would be "X Y" instead of just "X".</para>
-
-                <para><screen><varname>A</varname> = "1"
-<varname>A_append</varname> = "2"
-<varname>A_append</varname> = "3"
-<varname>A</varname> += "4"
-<varname>A</varname> .= "5"</screen></para>
-
-                <para>Would ultimately result in <varname>A</varname> taking the value "1 4523" since the _append operator executes at the same time as the expansion of other overrides.</para>
-
-            </section>
-            <section>
-                <title>Key Expansion</title>
-
-                <para>Key expansion happens at the data store finalisation time just before overrides are expanded.</para>
-
-                <para><screen><varname>A${B}</varname> = "X"
-<varname>B</varname> = "2"
-<varname>A2</varname> = "Y"</screen></para>
-                <para>So in this case <varname>A2</varname> would take the value of "X".</para>
-            </section>
-
-        </section>
-        <section>
-            <title>Dependency handling</title>
-            <para>BitBake 1.7.x onwards works with the metadata at the task level since this is optimal when dealing with multiple threads of execution. A robust method of specifing task dependencies is therefore needed. </para>
+            <title>Dependency Handling</title>
+            <para>Bitbake 1.7.x onwards works with the metadata at the task level since this is optimal when dealing with multiple threads of execution. A robust method of specifing task dependencies is therefore needed. </para>
             <section>
                 <title>Dependencies internal to the .bb file</title>
                 <para>Where the dependencies are internal to a given .bb file, the dependencies are handled by the previously detailed addtask directive.</para>
@@ -316,26 +249,26 @@ SRC_URI_append_1.0.7+ = "file://some_patch_which_the_new_versions_need.patch;pat
 
             <section>
                 <title>DEPENDS</title>
-                <para>DEPENDS lists build time dependencies. The 'deptask' flag for tasks is used to signify the task of each item listed in DEPENDS which must have completed before that task can be executed.</para>
+                <para>DEPENDS is taken to specify build time dependencies. The 'deptask' flag for tasks is used to signify the task of each DEPENDS which must have completed before that task can be executed.</para>
                 <para><screen>do_configure[deptask] = "do_populate_staging"</screen></para>
                 <para>means the do_populate_staging task of each item in DEPENDS must have completed before do_configure can execute.</para>
             </section>
             <section>
                 <title>RDEPENDS</title>
-                <para>RDEPENDS lists runtime dependencies. The 'rdeptask' flag for tasks is used to signify the task of each item listed in RDEPENDS which must have completed before that task can be executed.</para>
+                <para>RDEPENDS is taken to specify runtime dependencies. The 'rdeptask' flag for tasks is used to signify the task of each RDEPENDS which must have completed before that task can be executed.</para>
                 <para><screen>do_package_write[rdeptask] = "do_package"</screen></para>
                 <para>means the do_package task of each item in RDEPENDS must have completed before do_package_write can execute.</para>
             </section>
             <section>
                 <title>Recursive DEPENDS</title>
-                <para>These are specified with the 'recdeptask' flag and is used signify the task(s) of each DEPENDS which must have completed before that task can be executed. It applies recursively so the DEPENDS of each item in the original DEPENDS must be met and so on.</para>
+                <para>These are specified with the 'recdeptask' flag and is used signify the task(s) of each DEPENDS which must have completed before that task can be executed. It applies recursively so also, the DEPENDS of each item in the original DEPENDS must be met and so on.</para>
             </section>
             <section>
                 <title>Recursive RDEPENDS</title>
-                <para>These are specified with the 'recrdeptask' flag and is used signify the task(s) of each RDEPENDS which must have completed before that task can be executed. It applies recursively so the RDEPENDS of each item in the original RDEPENDS must be met and so on. It also runs all DEPENDS first.</para>
+                <para>These are specified with the 'recrdeptask' flag and is used signify the task(s) of each RDEPENDS which must have completed before that task can be executed. It applies recursively so also, the RDEPENDS of each item in the original RDEPENDS must be met and so on. It also runs all DEPENDS first too.</para>
             </section>
             <section>
-                <title>Inter task</title>
+                <title>Inter Task</title>
                 <para>The 'depends' flag for tasks is a more generic form of which allows an interdependency on specific tasks rather than specifying the data in DEPENDS or RDEPENDS.</para>
                 <para><screen>do_patch[depends] = "quilt-native:do_populate_staging"</screen></para>
                 <para>means the do_populate_staging task of the target quilt-native must have completed before the do_patch can execute.</para>
@@ -345,56 +278,35 @@ SRC_URI_append_1.0.7+ = "file://some_patch_which_the_new_versions_need.patch;pat
         <section>
             <title>Parsing</title>
             <section>
-                <title>Configuration files</title>
-                <para>The first kind of metadata in BitBake is configuration metadata.  This metadata is global, and therefore affects <emphasis>all</emphasis> packages and tasks which are executed.</para>
-                <para>BitBake will first search the current working directory for an optional "conf/bblayers.conf" configuration file. This file is expected to contain a BBLAYERS variable which is a space delimited list of 'layer' directories. For each directory in this list, a "conf/layer.conf" file will be searched for and parsed with the LAYERDIR variable being set to the directory where the layer was found. The idea is these files will setup BBPATH and other variables correctly for a given build directory automatically for the user.</para>
-                <para>BitBake will then expect to find 'conf/bitbake.conf' somewhere in the user specified <envar>BBPATH</envar>.  That configuration file generally has include directives to pull in any other metadata (generally files specific to architecture, machine, <emphasis>local</emphasis> and so on).</para>
+                <title>Configuration Files</title>
+                <para>The first of the classifications of metadata in BitBake is configuration metadata.  This metadata is global, and therefore affects <emphasis>all</emphasis> packages and tasks which are executed.</para>
+                <para>Bitbake will first search the current working directory for an optional "conf/bblayers.conf" configuration file. This file is expected to contain a BBLAYERS variable which is a space delimited list of 'layer' directories. For each directory in this list a "conf/layer.conf" file will be searched for and parsed with the LAYERDIR variable being set to the directory where the layer was found. The idea is these files will setup BBPATH and other variables correctly for a given build directory automatically for the user.</para>
+                <para>Bitbake will then expect to find 'conf/bitbake.conf' somewhere in the user specified <envar>BBPATH</envar>.  That configuration file generally has include directives to pull in any other metadata (generally files specific to architecture, machine, <emphasis>local</emphasis> and so on.</para>
                 <para>Only variable definitions and include directives are allowed in .conf files.</para>
             </section>
             <section>
                 <title>Classes</title>
-                <para>BitBake classes are our rudimentary inheritance mechanism.  As briefly mentioned in the metadata introduction, they're parsed when an <literal>inherit</literal> directive is encountered, and they are located in classes/ relative to the directories in <envar>BBPATH</envar>.</para>
+                <para>BitBake classes are our rudimentary inheritance mechanism.  As briefly mentioned in the metadata introduction, they're parsed when an <literal>inherit</literal> directive is encountered, and they are located in classes/ relative to the dirs in <envar>BBPATH</envar>.</para>
             </section>
             <section>
-                <title>.bb files</title>
+                <title>.bb Files</title>
                 <para>A BitBake (.bb) file is a logical unit of tasks to be executed.  Normally this is a package to be built.  Inter-.bb dependencies are obeyed.  The files themselves are located via the <varname>BBFILES</varname> variable, which is set to a space separated list of .bb files, and does handle wildcards.</para>
             </section>
         </section>
     </chapter>
 
     <chapter>
-        <title>File download support</title>
+        <title>File Download support</title>
         <section>
             <title>Overview</title>
-            <para>BitBake provides support to download files this procedure is called fetching and it handled by the fetch and fetch2 modules. At this point the original fetch code is considered to be replaced by fetch2 and this manual only related to the fetch2 codebase.</para>
-
-            <para>The SRC_URI is normally used to tell BitBake which files to fetch. The next sections will describe the available fetchers and their options. Each fetcher honors a set of variables and per URI parameters separated by a <quote>;</quote> consisting of a key and a value. The semantics of the variables and parameters are defined by the fetcher. BitBake tries to have consistent semantics between the different fetchers.
+            <para>BitBake provides support to download files this procedure is called fetching. The SRC_URI is normally used to indicate BitBake which files to fetch. The next sections will describe th available fetchers and the options they have. Each Fetcher honors a set of Variables and
+a per URI parameters separated by a <quote>;</quote> consisting of a key and a value. The semantic of the Variables and Parameters are defined by the Fetcher. BitBakes tries to have a consistent semantic between the different Fetchers.
             </para>
-
-            <para>The overall fetch process is that first, fetches are attempted from PREMIRRORS. If those don't work, the original SRC_URI is attempted and if that fails, BitBake will fall back to MIRRORS. Cross urls are supported, so its possible to mirror a git repository on an http server as a tarball for example. Some example commonly used mirror definitions are:</para>
-
-            <para><screen><varname>PREMIRRORS</varname> ?= "\
-bzr://.*/.*   http://somemirror.org/sources/ \n \
-cvs://.*/.*   http://somemirror.org/sources/ \n \
-git://.*/.*   http://somemirror.org/sources/ \n \
-hg://.*/.*    http://somemirror.org/sources/ \n \
-osc://.*/.*   http://somemirror.org/sources/ \n \
-p4://.*/.*    http://somemirror.org/sources/ \n \
-svk://.*/.*   http://somemirror.org/sources/ \n \
-svn://.*/.*   http://somemirror.org/sources/ \n"
-
-<varname>MIRRORS</varname> =+ "\
-ftp://.*/.*      http://somemirror.org/sources/ \n \
-http://.*/.*     http://somemirror.org/sources/ \n \
-https://.*/.*    http://somemirror.org/sources/ \n"</screen></para>
-
-            <para>Non-local downloaded output is placed into the directory specified by the <varname>DL_DIR</varname>. For non local downloads the code can check checksums for the download to ensure the file has been downloaded correctly. These are specified in the form <varname>SRC_URI[md5sum]</varname> for the md5 checksum and <varname>SRC_URI[sha256sum]</varname> for the sha256 checksum. If <varname>BB_STRICT_CHECKSUM</varname> is set, any download without a checksum will trigger an error message. In cases where multiple files are listed in SRC_URI, the name parameter is used assign names to the urls and these are then specified in the checksums in the form SRC_URI[name.sha256sum].</para>
-
         </section>
 
         <section>
-            <title>Local file fetcher</title>
-            <para>The URN for the local file fetcher is <emphasis>file</emphasis>. The filename can be either absolute or relative. If the filename is relative, <varname>FILESPATH</varname> and failing that <varname>FILESDIR</varname> will be used to find the appropriate relative file. The metadata usually extend these variables to include variations of the values in <varname>OVERRIDES</varname>. Single files and complete directories can be specified.
+            <title>Local File Fetcher</title>
+            <para>The URN for the Local File Fetcher is <emphasis>file</emphasis>. The filename can be either absolute or relative. If the filename is relative <varname>FILESPATH</varname> and <varname>FILESDIR</varname> will be used to find the appropriate relative file depending on the <varname>OVERRIDES</varname>. Single files and complete directories can be specified.
 <screen><varname>SRC_URI</varname>= "file://relativefile.patch"
 <varname>SRC_URI</varname>= "file://relativefile.patch;this=ignored"
 <varname>SRC_URI</varname>= "file:///Users/ich/very_important_software"
@@ -403,11 +315,10 @@ https://.*/.*    http://somemirror.org/sources/ \n"</screen></para>
         </section>
 
         <section>
-            <title>CVS fetcher</title>
-            <para>The URN for the CVS fetcher is <emphasis>cvs</emphasis>. This fetcher honors the variables <varname>CVSDIR</varname>, <varname>SRCDATE</varname>, <varname>FETCHCOMMAND_cvs</varname>, <varname>UPDATECOMMAND_cvs</varname>. <varname>DL_DIR</varname> specifies where a temporary checkout is saved. <varname>SRCDATE</varname> specifies which date to use when doing the fetching (the special value of "now" will cause the checkout to be updated on every build). <varname>FETCHCOMMAND</varname> and <varname>UPDATECOMMAND</varname> specify which executables to use for the CVS checkout or update.
+            <title>CVS File Fetcher</title>
+            <para>The URN for the CVS Fetcher is <emphasis>cvs</emphasis>. This Fetcher honors the variables <varname>DL_DIR</varname>, <varname>SRCDATE</varname>, <varname>FETCHCOMMAND_cvs</varname>, <varname>UPDATECOMMAND_cvs</varname>. <varname>DL_DIR</varname> specifies where a temporary checkout is saved, <varname>SRCDATE</varname> specifies which date to use when doing the fetching (the special value of "now" will cause the checkout to be updated on every build), <varname>FETCHCOMMAND</varname> and <varname>UPDATECOMMAND</varname> specify which executables should be used when doing the CVS checkout or update.
             </para>
-            <para>The supported parameters are <varname>module</varname>, <varname>tag</varname>, <varname>date</varname>, <varname>method</varname>, <varname>localdir</varname>, <varname>rsh</varname> and <varname>scmdata</varname>. The <varname>module</varname> specifies which module to check out, the <varname>tag</varname> describes which CVS TAG should be used for the checkout. By default the TAG is empty. A <varname>date</varname> can be specified to override the SRCDATE of the configuration to checkout a specific date.  The special value of "now" will cause the checkout to be updated on every build.<varname>method</varname> is by default <emphasis>pserver</emphasis>. If <emphasis>ext</emphasis> is used the <varname>rsh</varname> parameter will be evaluated and <varname>CVS_RSH</varname> will be set. Finally, <varname>localdir</varname> is used to checkout into a special directory relative to <varname>CVSDIR</varname>.
-
+            <para>The supported Parameters are <varname>module</varname>, <varname>tag</varname>, <varname>date</varname>, <varname>method</varname>, <varname>localdir</varname>, <varname>rsh</varname>. The <varname>module</varname> specifies which module to check out, the <varname>tag</varname> describes which CVS TAG should be used for the checkout by default the TAG is empty. A <varname>date</varname> can be specified to override the SRCDATE of the configuration to checkout a specific date.  The special value of "now" will cause the checkout to be updated on every build.<varname>method</varname> is by default <emphasis>pserver</emphasis>, if <emphasis>ext</emphasis> is used the <varname>rsh</varname> parameter will be evaluated and <varname>CVS_RSH</varname> will be set. Finally <varname>localdir</varname> is used to checkout into a special directory relative to <varname>CVSDIR</varname>.
 <screen><varname>SRC_URI</varname> = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext"
 <varname>SRC_URI</varname> = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat"
 </screen>
@@ -415,22 +326,32 @@ https://.*/.*    http://somemirror.org/sources/ \n"</screen></para>
         </section>
 
         <section>
-            <title>HTTP/FTP fetcher</title>
-            <para>The URNs for the HTTP/FTP fetcher are <emphasis>http</emphasis>, <emphasis>https</emphasis> and <emphasis>ftp</emphasis>. This fetcher honors the variables <varname>FETCHCOMMAND_wget</varname>. <varname>FETCHCOMMAND</varname> contains the command used for fetching. <quote>${URI}</quote> and <quote>${FILES}</quote> will be replaced by the URI and basename of the file to be fetched.
+            <title>HTTP/FTP Fetcher</title>
+            <para>The URNs for the HTTP/FTP are <emphasis>http</emphasis>, <emphasis>https</emphasis> and <emphasis>ftp</emphasis>. This Fetcher honors the variables <varname>DL_DIR</varname>, <varname>FETCHCOMMAND_wget</varname>, <varname>PREMIRRORS</varname>, <varname>MIRRORS</varname>. The <varname>DL_DIR</varname> defines where to store the fetched file, <varname>FETCHCOMMAND</varname> contains the command used for fetching. <quote>${URI}</quote> and <quote>${FILES}</quote> will be replaced by the uri and basename of the to be fetched file. <varname>PREMIRRORS</varname>
+will be tried first when fetching a file if that fails the actual file will be tried and finally all <varname>MIRRORS</varname> will be tried.
             </para>
-            <para><screen><varname>SRC_URI</varname> = "http://oe.handhelds.org/not_there.aac"
-<varname>SRC_URI</varname> = "ftp://oe.handhelds.org/not_there_as_well.aac"
-<varname>SRC_URI</varname> = "ftp://you@oe.handheld.sorg/home/you/secret.plan"
+            <para>The only supported Parameter is <varname>md5sum</varname>. After a fetch the <varname>md5sum</varname> of the file will be calculated and the two sums will be compared.
+            </para>
+            <para><screen><varname>SRC_URI</varname> = "http://oe.handhelds.org/not_there.aac;md5sum=12343"
+<varname>SRC_URI</varname> = "ftp://oe.handhelds.org/not_there_as_well.aac;md5sum=1234"
+<varname>SRC_URI</varname> = "ftp://you@oe.handheld.sorg/home/you/secret.plan;md5sum=1234"
 </screen></para>
         </section>
 
         <section>
-            <title>SVN fetcher</title>
-            <para>The URN for the SVN fetcher is <emphasis>svn</emphasis>.
+            <title>SVK Fetcher</title>
+            <para>
+            <emphasis>Currently NOT supported</emphasis>
             </para>
-            <para>This fetcher honors the variables <varname>FETCHCOMMAND_svn</varname>, <varname>SVNDIR</varname>, <varname>SRCREV</varname>. <varname>FETCHCOMMAND</varname> contains the subversion command. <varname>SRCREV</varname> specifies which revision to use when doing the fetching.
+        </section>
+
+        <section>
+            <title>SVN Fetcher</title>
+            <para>The URN for the SVN Fetcher is <emphasis>svn</emphasis>.
             </para>
-            <para>The supported parameters are <varname>proto</varname>, <varname>rev</varname> and <varname>scmdata</varname>. <varname>proto</varname> is the Subversion protocol, <varname>rev</varname> is the Subversion revision. If <varname>scmdata</varname> is set to <quote>keep</quote>, the <quote>.svn</quote> directories will be available during compile-time.
+            <para>This Fetcher honors the variables <varname>FETCHCOMMAND_svn</varname>, <varname>DL_DIR</varname>, <varname>SRCDATE</varname>. <varname>FETCHCOMMAND</varname> contains the subversion command, <varname>DL_DIR</varname> is the directory where tarballs will be saved, <varname>SRCDATE</varname> specifies which date to use when doing the fetching (the special value of "now" will cause the checkout to be updated on every build).
+            </para>
+            <para>The supported Parameters are <varname>proto</varname>, <varname>rev</varname>. <varname>proto</varname> is the subversion prototype, <varname>rev</varname> is the subversions revision.
             </para>
             <para><screen><varname>SRC_URI</varname> = "svn://svn.oe.handhelds.org/svn;module=vip;proto=http;rev=667"
 <varname>SRC_URI</varname> = "svn://svn.oe.handhelds.org/svn/;module=opie;proto=svn+ssh;date=20060126"
@@ -438,12 +359,12 @@ https://.*/.*    http://somemirror.org/sources/ \n"</screen></para>
         </section>
 
         <section>
-            <title>GIT fetcher</title>
+            <title>GIT Fetcher</title>
             <para>The URN for the GIT Fetcher is <emphasis>git</emphasis>.
             </para>
-            <para>The variable <varname>GITDIR</varname> will be used as the base directory where the git tree is cloned to.
+            <para>The Variables <varname>DL_DIR</varname>, <varname>GITDIR</varname> are used. <varname>DL_DIR</varname> will be used to store the checkedout version. <varname>GITDIR</varname> will be used as the base directory where the git tree is cloned to.
             </para>
-            <para>The parameters are <emphasis>tag</emphasis>, <emphasis>protocol</emphasis> and <emphasis>scmdata</emphasis>. <emphasis>tag</emphasis> is a Git tag, the default is <quote>master</quote>. <emphasis>protocol</emphasis> is the Git protocol to use and defaults to <quote>git</quote> if a hostname is set, otherwise its <quote>file</quote>. If <emphasis>scmdata</emphasis> is set to <quote>keep</quote>, the <quote>.git</quote> directory will be available during compile-time.
+            <para>The Parameters are <emphasis>tag</emphasis>, <emphasis>protocol</emphasis>. <emphasis>tag</emphasis> is a git tag, the default is <quote>master</quote>. <emphasis>protocol</emphasis> is the git protocol to use and defaults to <quote>rsync</quote>.
             </para>
             <para><screen><varname>SRC_URI</varname> = "git://git.oe.handhelds.org/git/vip.git;tag=version-1"
 <varname>SRC_URI</varname> = "git://git.oe.handhelds.org/git/vip.git;protocol=http"
@@ -454,13 +375,13 @@ https://.*/.*    http://somemirror.org/sources/ \n"</screen></para>
 
 
     <chapter>
-        <title>The BitBake command</title>
+        <title>The bitbake command</title>
             <section>
                 <title>Introduction</title>
                 <para>bitbake is the primary command in the system.  It facilitates executing tasks in a single .bb file, or executing a given task on a set of multiple .bb files, accounting for interdependencies amongst them.</para>
             </section>
             <section>
-                <title>Usage and syntax</title>
+                <title>Usage and Syntax</title>
                 <para>
                     <screen><prompt>$ </prompt>bitbake --help
 usage: bitbake [options] [package ...]
@@ -496,6 +417,8 @@ options:
                         than once.
   -n, --dry-run         don't execute, just go through the motions
   -p, --parse-only      quit after parsing the BB files (developers only)
+  -d, --disable-psyco   disable using the psyco just-in-time compiler (not
+                        recommended)
   -s, --show-versions   show current and preferred versions of all packages
   -e, --environment     show the global or per-package environment (this is
                         what used to be bbread)
@@ -515,7 +438,7 @@ options:
                 <para>
                 <example>
                     <title>Executing a task against a single .bb</title>
-                    <para>Executing tasks for a single file is relatively simple.  You specify the file in question, and BitBake parses it and executes the specified task (or <quote>build</quote> by default).  It obeys intertask dependencies when doing so.</para>
+                    <para>Executing tasks for a single file is relatively simple.  You specify the file in question, and bitbake parses it and executes the specified task (or <quote>build</quote> by default).  It obeys intertask dependencies when doing so.</para>
                     <para><quote>clean</quote> task:</para>
                     <para><screen><prompt>$ </prompt>bitbake -b blah_1.0.bb -c clean</screen></para>
                     <para><quote>build</quote> task:</para>
@@ -525,8 +448,8 @@ options:
                 <para>
                 <example>
                     <title>Executing tasks against a set of .bb files</title>
-                    <para>There are a number of additional complexities introduced when one wants to manage multiple .bb files.  Clearly there needs to be a way to tell BitBake what files are available, and of those, which we want to execute at this time.  There also needs to be a way for each .bb to express its dependencies, both for build time and runtime.  There must be a way for the user to express their preferences when multiple .bb's provide the same functionality, or when there are multiple versions of a .bb.</para>
-                    <para>The next section, Metadata, outlines how to specify such things.</para>
+                    <para>There are a number of additional complexities introduced when one wants to manage multiple .bb files.  Clearly there needs to be a way to tell bitbake what files are available, and of those, which we want to execute at this time.  There also needs to be a way for each .bb to express its dependencies, both for build time and runtime.  There must be a way for the user to express their preferences when multiple .bb's provide the same functionality, or when there are multiple versions of a .bb.</para>
+                    <para>The next section, Metadata, outlines how one goes about specifying such things.</para>
                     <para>Note that the bitbake command, when not using --buildfile, accepts a <varname>PROVIDER</varname>, not a filename or anything else.  By default, a .bb generally PROVIDES its packagename, packagename-version, and packagename-version-revision.</para>
                     <screen><prompt>$ </prompt>bitbake blah</screen>
                     <screen><prompt>$ </prompt>bitbake blah-1.0</screen>
@@ -538,8 +461,8 @@ options:
                 <example>
                     <title>Generating dependency graphs</title>
                     <para>BitBake is able to generate dependency graphs using the dot syntax. These graphs can be converted
-to images using the <application>dot</application> application from <ulink url="http://www.graphviz.org">Graphviz</ulink>. 
-Two files will be written into the current working directory, <emphasis>depends.dot</emphasis> containing dependency information at the package level and <emphasis>task-depends.dot</emphasis> containing a breakdown of the dependencies at the task level. To stop depending on common depends, one can use the <prompt>-I depend</prompt> to omit these from the graph. This can lead to more readable graphs.  This way, <varname>DEPENDS</varname> from inherited classes such as base.bbclass can be removed from the graph.</para>
+to images using the <application>dot</application> application from <ulink url="http://www.graphviz.org">graphviz</ulink>. 
+Two files will be written into the current working directory, <emphasis>depends.dot</emphasis> containing dependency information at the package level and <emphasis>task-depends.dot</emphasis> containing a breakdown of the dependencies at the task level. To stop depending on common depends one can use the <prompt>-I depend</prompt> to omit these from the graph. This can lead to more readable graphs. E.g. this way <varname>DEPENDS</varname> from inherited classes, e.g. base.bbclass, can be removed from the graph.</para>
                     <screen><prompt>$ </prompt>bitbake -g blah</screen>
                     <screen><prompt>$ </prompt>bitbake -g -I virtual/whatever -I bloom blah</screen>
                 </example>
@@ -547,20 +470,20 @@ Two files will be written into the current working directory, <emphasis>depends.
             </section>
             <section>
                 <title>Special variables</title>
-                <para>Certain variables affect BitBake operation:</para>
+                <para>Certain variables affect bitbake operation:</para>
                 <section>
                     <title><varname>BB_NUMBER_THREADS</varname></title>
-                    <para> The number of threads BitBake should run at once (default: 1).</para>
+                    <para> The number of threads bitbake should run at once (default: 1).</para>
                 </section>
             </section>
             <section>
                 <title>Metadata</title>
-                <para>As you may have seen in the usage information, or in the information about .bb files, the <varname>BBFILES</varname> variable is how the BitBake tool locates its files.  This variable is a space separated list of files that are available, and supports wildcards.
+                <para>As you may have seen in the usage information, or in the information about .bb files, the BBFILES variable is how the bitbake tool locates its files.  This variable is a space separated list of files that are available, and supports wildcards.
                 <example>
                     <title>Setting BBFILES</title>
                     <programlisting><varname>BBFILES</varname> = "/path/to/bbfiles/*.bb"</programlisting>
                 </example></para>
-                <para>With regard to dependencies, it expects the .bb to define a <varname>DEPENDS</varname> variable, which contains a space separated list of <quote>package names</quote>, which themselves are the <varname>PN</varname> variable.  The <varname>PN</varname> variable is, in general, set to a component of the .bb filename by default.</para>
+                <para>With regard to dependencies, it expects the .bb to define a <varname>DEPENDS</varname> variable, which contains a space separated list of <quote>package names</quote>, which themselves are the <varname>PN</varname> variable.  The <varname>PN</varname> variable is, in general, by default, set to a component of the .bb filename.</para>
                 <example>
                     <title>Depending on another .bb</title>
                     <para>a.bb:
@@ -573,7 +496,7 @@ DEPENDS += "package-b"</screen>
                 </example>
                 <example>
                     <title>Using PROVIDES</title>
-                    <para>This example shows the usage of the <varname>PROVIDES</varname> variable, which allows a given .bb to specify what functionality it provides.</para>
+                    <para>This example shows the usage of the PROVIDES variable, which allows a given .bb to specify what functionality it provides.</para>
                     <para>package1.bb:
     <screen>PROVIDES += "virtual/package"</screen>
                     </para>
@@ -583,16 +506,16 @@ DEPENDS += "package-b"</screen>
                     <para>package3.bb:
     <screen>PROVIDES += "virtual/package"</screen>
                     </para>
-                    <para>As you can see, we have two different .bb's that provide the same functionality (virtual/package).  Clearly, there needs to be a way for the person running BitBake to control which of those providers gets used.  There is, indeed, such a way.</para>
+                    <para>As you can see, here there are two different .bb's that provide the same functionality (virtual/package).  Clearly, there needs to be a way for the person running bitbake to control which of those providers gets used.  There is, indeed, such a way.</para>
                     <para>The following would go into a .conf file, to select package1:
     <screen>PREFERRED_PROVIDER_virtual/package = "package1"</screen>
                     </para>
                 </example>
                 <example>
                     <title>Specifying version preference</title>
-                    <para>When there are multiple <quote>versions</quote> of a given package, BitBake defaults to selecting the most recent version, unless otherwise specified.  If the .bb in question has a <varname>DEFAULT_PREFERENCE</varname> set lower than the other .bb's (default is 0), then it will not be selected.  This allows the person or persons maintaining the repository of .bb files to specify their preference for the default selected version.  In addition, the user can specify their preferred version.</para>
+                    <para>When there are multiple <quote>versions</quote> of a given package, bitbake defaults to selecting the most recent version, unless otherwise specified.  If the .bb in question has a <varname>DEFAULT_PREFERENCE</varname> set lower than the other .bb's (default is 0), then it will not be selected.  This allows the person or persons maintaining the repository of .bb files to specify their preferences for the default selected version.  In addition, the user can specify their preferences with regard to version.</para>
                     <para>If the first .bb is named <filename>a_1.1.bb</filename>, then the <varname>PN</varname> variable will be set to <quote>a</quote>, and the <varname>PV</varname> variable will be set to 1.1.</para>
-                    <para>If we then have an <filename>a_1.2.bb</filename>, BitBake will choose 1.2 by default.  However, if we define the following variable in a .conf that BitBake parses, we can change that.
+                    <para>If we then have an <filename>a_1.2.bb</filename>, bitbake will choose 1.2 by default.  However, if we define the following variable in a .conf that bitbake parses, we can change that.
     <screen>PREFERRED_VERSION_a = "1.1"</screen>
                     </para>
                 </example>

bitbake/lib/bb/__init__.py

@@ -21,45 +21,13 @@
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
-__version__ = "1.13.3"
+__version__ = "1.11.0"
 
 import sys
 if sys.version_info < (2, 6, 0):
     raise RuntimeError("Sorry, python 2.6.0 or later is required for this version of bitbake")
 
 import os
-import logging
-
-
-class NullHandler(logging.Handler):
-    def emit(self, record):
-        pass
-
-Logger = logging.getLoggerClass()
-class BBLogger(Logger):
-    def __init__(self, name):
-        if name.split(".")[0] == "BitBake":
-            self.debug = self.bbdebug
-        Logger.__init__(self, name)
-
-    def bbdebug(self, level, msg, *args, **kwargs):
-        return self.log(logging.DEBUG - level + 1, msg, *args, **kwargs)
-
-    def plain(self, msg, *args, **kwargs):
-        return self.log(logging.INFO + 1, msg, *args, **kwargs)
-
-    def verbose(self, msg, *args, **kwargs):
-        return self.log(logging.INFO - 1, msg, *args, **kwargs)
-
-logging.raiseExceptions = False
-logging.setLoggerClass(BBLogger)
-
-logger = logging.getLogger("BitBake")
-logger.addHandler(NullHandler())
-logger.setLevel(logging.DEBUG - 2)
-
-# This has to be imported after the setLoggerClass, as the import of bb.msg
-# can result in construction of the various loggers.
 import bb.msg
 
 if "BBDEBUG" in os.environ:
@@ -67,36 +35,28 @@ if "BBDEBUG" in os.environ:
     if level:
         bb.msg.set_debug_level(level)
 
-if True or os.environ.get("BBFETCH2"):
-    from bb import fetch2 as fetch
-    sys.modules['bb.fetch'] = sys.modules['bb.fetch2']
 
 # Messaging convenience functions
 def plain(*args):
-    logger.plain(''.join(args))
+    bb.msg.plain(''.join(args))
 
 def debug(lvl, *args):
-    if isinstance(lvl, basestring):
-        logger.warn("Passed invalid debug level '%s' to bb.debug", lvl)
-        args = (lvl,) + args
-        lvl = 1
-    logger.debug(lvl, ''.join(args))
+    bb.msg.debug(lvl, None, ''.join(args))
 
 def note(*args):
-    logger.info(''.join(args))
+    bb.msg.note(1, None, ''.join(args))
 
 def warn(*args):
-    logger.warn(''.join(args))
+    bb.msg.warn(None, ''.join(args))
 
 def error(*args):
-    logger.error(''.join(args))
+    bb.msg.error(None, ''.join(args))
 
 def fatal(*args):
-    logger.critical(''.join(args))
-    sys.exit(1)
+    bb.msg.fatal(None, ''.join(args))
 
 
-def deprecated(func, name=None, advice=""):
+def deprecated(func, name = None, advice = ""):
     """This is a decorator which can be used to mark functions
     as deprecated. It will result in a warning being emmitted
     when the function is used."""
@@ -110,8 +70,8 @@ def deprecated(func, name=None, advice=""):
     def newFunc(*args, **kwargs):
         warnings.warn("Call to deprecated function %s%s." % (name,
                                                              advice),
-                      category=DeprecationWarning,
-                      stacklevel=2)
+                      category = PendingDeprecationWarning,
+                      stacklevel = 2)
         return func(*args, **kwargs)
     newFunc.__name__ = func.__name__
     newFunc.__doc__ = func.__doc__

bitbake/lib/bb/build.py

@@ -25,21 +25,9 @@
 #
 #Based on functions from the base bb module, Copyright 2003 Holger Schurig
 
-import os
-import sys
-import logging
-import shlex
-import bb
-import bb.msg
-import bb.process
-from contextlib import nested
-from bb import data, event, utils
-
-bblogger = logging.getLogger('BitBake')
-logger = logging.getLogger('BitBake.Build')
-
-NULL = open(os.devnull, 'r+')
-
+from bb import data, event, mkdirhier, utils
+import bb, os, sys
+import bb.utils
 
 # When we execute a python function we'd like certain things
 # in all namespaces, hence we add them to __builtins__
@@ -48,22 +36,13 @@ NULL = open(os.devnull, 'r+')
 __builtins__['bb'] = bb
 __builtins__['os'] = os
 
+# events
 class FuncFailed(Exception):
-    def __init__(self, name = None, logfile = None):
-        self.logfile = logfile
-        self.name = name
-        if name:
-            self.msg = "Function '%s' failed" % name
-        else:
-            self.msg = "Function failed"
-
-    def __str__(self):
-        if self.logfile and os.path.exists(self.logfile):
-            msg = ("%s (see %s for further information)" %
-                   (self.msg, self.logfile))
-        else:
-            msg = self.msg
-        return msg
+    """
+    Executed function failed
+    First parameter a message
+    Second paramter is a logfile (optional)
+    """
 
 class TaskBase(event.Event):
     """Base class for task events"""
@@ -90,56 +69,38 @@ class TaskSucceeded(TaskBase):
 
 class TaskFailed(TaskBase):
     """Task execution failed"""
-
-    def __init__(self, task, logfile, metadata):
+    def __init__(self, msg, logfile, t, d ):
         self.logfile = logfile
-        super(TaskFailed, self).__init__(task, metadata)
+        self.msg = msg
+        TaskBase.__init__(self, t, d)
 
 class TaskInvalid(TaskBase):
+    """Invalid Task"""
 
-    def __init__(self, task, metadata):
-        super(TaskInvalid, self).__init__(task, metadata)
-        self._message = "No such task '%s'" % task
-
-
-class LogTee(object):
-    def __init__(self, logger, outfile):
-        self.outfile = outfile
-        self.logger = logger
-        self.name = self.outfile.name
-
-    def write(self, string):
-        self.logger.plain(string)
-        self.outfile.write(string)
-
-    def __enter__(self):
-        self.outfile.__enter__()
-        return self
-
-    def __exit__(self, *excinfo):
-        self.outfile.__exit__(*excinfo)
-
-    def __repr__(self):
-        return '<LogTee {0}>'.format(self.name)
-
+# functions
 
 def exec_func(func, d, dirs = None):
     """Execute an BB 'function'"""
 
     body = data.getVar(func, d)
     if not body:
-        if body is None:
-            logger.warn("Function %s doesn't exist", func)
+        bb.warn("Function %s doesn't exist" % func)
         return
 
     flags = data.getVarFlags(func, d)
-    cleandirs = flags.get('cleandirs')
+    for item in ['deps', 'check', 'interactive', 'python', 'cleandirs', 'dirs', 'lockfiles', 'fakeroot', 'task']:
+        if not item in flags:
+            flags[item] = None
+
+    ispython = flags['python']
+
+    cleandirs = flags['cleandirs']
     if cleandirs:
         for cdir in data.expand(cleandirs, d).split():
-            bb.utils.remove(cdir, True)
+            os.system("rm -rf %s" % cdir)
 
     if dirs is None:
-        dirs = flags.get('dirs')
+        dirs = flags['dirs']
         if dirs:
             dirs = data.expand(dirs, d).split()
 
@@ -149,255 +110,277 @@ def exec_func(func, d, dirs = None):
         adir = dirs[-1]
     else:
         adir = data.getVar('B', d, 1)
-        bb.utils.mkdirhier(adir)
 
-    ispython = flags.get('python')
-    if flags.get('fakeroot') and not flags.get('task'):
-        bb.fatal("Function %s specifies fakeroot but isn't a task?!" % func)
+    # Save current directory
+    try:
+        prevdir = os.getcwd()
+    except OSError:
+        prevdir = data.getVar('TOPDIR', d, True)
 
-    lockflag = flags.get('lockfiles')
-    if lockflag:
-        lockfiles = [data.expand(f, d) for f in lockflag.split()]
-    else:
-        lockfiles = None
+    # Setup scriptfile
+    t = data.getVar('T', d, 1)
+    if not t:
+        raise SystemExit("T variable not set, unable to build")
+    bb.utils.mkdirhier(t)
+    runfile = "%s/run.%s.%s" % (t, func, str(os.getpid()))
+    logfile = d.getVar("BB_LOGFILE", True)
 
-    tempdir = data.getVar('T', d, 1)
-    bb.utils.mkdirhier(tempdir)
-    runfile = os.path.join(tempdir, 'run.{0}.{1}'.format(func, os.getpid()))
+    # Change to correct directory (if specified)
+    if adir and os.access(adir, os.F_OK):
+        os.chdir(adir)
 
-    with bb.utils.fileslocked(lockfiles):
-        if ispython:
-            exec_func_python(func, d, runfile, cwd=adir)
-        else:
-            exec_func_shell(func, d, runfile, cwd=adir)
-
-_functionfmt = """
-def {function}(d):
-{body}
-
-{function}(d)
-"""
-logformatter = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
-def exec_func_python(func, d, runfile, cwd=None):
-    """Execute a python BB 'function'"""
-
-    bbfile = d.getVar('FILE', True)
-    code = _functionfmt.format(function=func, body=d.getVar(func, True))
-    bb.utils.mkdirhier(os.path.dirname(runfile))
-    with open(runfile, 'w') as script:
-        script.write(code)
-
-    if cwd:
-        try:
-            olddir = os.getcwd()
-        except OSError:
-            olddir = None
-        os.chdir(cwd)
+    locks = []
+    lockfiles = flags['lockfiles']
+    if lockfiles:
+        for lock in data.expand(lockfiles, d).split():
+            locks.append(bb.utils.lockfile(lock))
 
     try:
-        comp = utils.better_compile(code, func, bbfile)
-        utils.better_exec(comp, {"d": d}, code, bbfile)
-    except:
-        if sys.exc_info()[0] in (bb.parse.SkipPackage, bb.build.FuncFailed):
-            raise
+        # Run the function
+        if ispython:
+            exec_func_python(func, d, runfile, logfile)
+        else:
+            exec_func_shell(func, d, runfile, logfile, flags)
+
+        # Restore original directory
+        try:
+            os.chdir(prevdir)
+        except:
+            pass
 
-        raise FuncFailed(func, None)
     finally:
-        if cwd and olddir:
-            try:
-                os.chdir(olddir)
-            except OSError:
-                pass
 
-def exec_func_shell(function, d, runfile, cwd=None):
-    """Execute a shell function from the metadata
+        # Unlock any lockfiles
+        for lock in locks:
+            bb.utils.unlockfile(lock)
+
+def exec_func_python(func, d, runfile, logfile):
+    """Execute a python BB 'function'"""
+
+    bbfile = bb.data.getVar('FILE', d, 1)
+    tmp  = "def " + func + "(d):\n%s" % data.getVar(func, d)
+    tmp += '\n' + func + '(d)'
+
+    f = open(runfile, "w")
+    f.write(tmp)
+    comp = utils.better_compile(tmp, func, bbfile)
+    try:
+        utils.better_exec(comp, {"d": d}, tmp, bbfile)
+    except:
+        (t, value, tb) = sys.exc_info()
+
+        if t in [bb.parse.SkipPackage, bb.build.FuncFailed]:
+            raise
+        raise FuncFailed("Function %s failed" % func, logfile)
+
+
+def exec_func_shell(func, d, runfile, logfile, flags):
+    """Execute a shell BB 'function' Returns true if execution was successful.
+
+    For this, it creates a bash shell script in the tmp dectory, writes the local
+    data into it and finally executes. The output of the shell will end in a log file and stdout.
 
     Note on directory behavior.  The 'dirs' varflag should contain a list
     of the directories you need created prior to execution.  The last
     item in the list is where we will chdir/cd to.
     """
 
-    # Don't let the emitted shell script override PWD
-    d.delVarFlag('PWD', 'export')
+    deps = flags['deps']
+    check = flags['check']
+    if check in globals():
+        if globals()[check](func, deps):
+            return
 
-    with open(runfile, 'w') as script:
-        script.write('#!/bin/sh -e\n')
-        if bb.msg.loggerDefaultVerbose:
-            script.write("set -x\n")
-        data.emit_func(function, script, d)
-        if cwd:
-            script.write("cd %s\n" % cwd)
-        script.write("%s\n" % function)
+    f = open(runfile, "w")
+    f.write("#!/bin/sh -e\n")
+    if bb.msg.debug_level['default'] > 0: f.write("set -x\n")
+    data.emit_func(func, f, d)
 
+    f.write("cd %s\n" % os.getcwd())
+    if func: f.write("%s\n" % func)
+    f.close()
     os.chmod(runfile, 0775)
+    if not func:
+        raise FuncFailed("Function not specified for exec_func_shell")
 
-    cmd = runfile
+    # execute function
+    if flags['fakeroot'] and not flags['task']:
+        bb.fatal("Function %s specifies fakeroot but isn't a task?!" % func)
 
-    if bb.msg.loggerDefaultVerbose:
-        logfile = LogTee(logger, sys.stdout)
-    else:
-        logfile = sys.stdout
+    lang_environment = "LC_ALL=C "
+    ret = os.system('%ssh -e %s' % (lang_environment, runfile))
 
-    try:
-        bb.process.run(cmd, shell=False, stdin=NULL, log=logfile)
-    except bb.process.CmdError:
-        logfn = d.getVar('BB_LOGFILE', True)
-        raise FuncFailed(function, logfn)
+    if ret == 0:
+        return
 
-def _task_data(fn, task, d):
-    localdata = data.createCopy(d)
-    localdata.setVar('BB_FILENAME', fn)
-    localdata.setVar('BB_CURRENTTASK', task[3:])
-    localdata.setVar('OVERRIDES', 'task-%s:%s' %
-                     (task[3:], d.getVar('OVERRIDES', False)))
-    localdata.finalize()
-    data.expandKeys(localdata)
-    return localdata
+    raise FuncFailed("function %s failed" % func, logfile)
 
-def _exec_task(fn, task, d, quieterr):
-    """Execute a BB 'task'
 
-    Execution of a task involves a bit more setup than executing a function,
-    running it with its own local metadata, and with some useful variables set.
-    """
+def exec_task(fn, task, d):
+    """Execute an BB 'task'
+
+       The primary difference between executing a task versus executing
+       a function is that a task exists in the task digraph, and therefore
+       has dependencies amongst other tasks."""
+
+    # Check whther this is a valid task
     if not data.getVarFlag(task, 'task', d):
         event.fire(TaskInvalid(task, d), d)
-        logger.error("No such task: %s" % task)
+        bb.msg.error(bb.msg.domain.Build, "No such task: %s" % task)
         return 1
 
-    logger.debug(1, "Executing task %s", task)
+    quieterr = False
+    if d.getVarFlag(task, "quieterrors") is not None:
+         quieterr = True
 
-    localdata = _task_data(fn, task, d)
-    tempdir = localdata.getVar('T', True)
-    if not tempdir:
-        bb.fatal("T variable not set, unable to build")
+    try:
+        bb.msg.debug(1, bb.msg.domain.Build, "Executing task %s" % task)
+        old_overrides = data.getVar('OVERRIDES', d, 0)
+        localdata = data.createCopy(d)
+        data.setVar('OVERRIDES', 'task-%s:%s' % (task[3:], old_overrides), localdata)
+        data.update_data(localdata)
+        data.expandKeys(localdata)
+        data.setVar('BB_FILENAME', fn, d)
+        data.setVar('BB_CURRENTTASK', task[3:], d)
+        event.fire(TaskStarted(task, localdata), localdata)
 
-    bb.utils.mkdirhier(tempdir)
-    loglink = os.path.join(tempdir, 'log.{0}'.format(task))
-    logfn = os.path.join(tempdir, 'log.{0}.{1}'.format(task, os.getpid()))
-    if loglink:
-        bb.utils.remove(loglink)
+        # Setup logfiles
+        t = data.getVar('T', d, 1)
+        if not t:
+            raise SystemExit("T variable not set, unable to build")
+        bb.utils.mkdirhier(t)
+        loglink = "%s/log.%s" % (t, task)
+        logfile = "%s/log.%s.%s" % (t, task, str(os.getpid()))
+        d.setVar("BB_LOGFILE", logfile)
 
+        # Even though the log file has not yet been opened, lets create the link
+        if loglink:
+            try:
+                os.remove(loglink)
+            except OSError as e:
+                pass
+
+            try:
+                os.symlink(logfile, loglink)
+            except OSError as e:
+                pass
+
+        # Handle logfiles
+        si = file('/dev/null', 'r')
         try:
-           os.symlink(logfn, loglink)
-        except OSError:
-           pass
+            so = file(logfile, 'w')
+        except OSError as e:
+            bb.msg.error(bb.msg.domain.Build, "opening log file: %s" % e)
+            pass
+        se = so
 
-    prefuncs = localdata.getVarFlag(task, 'prefuncs', expand=True)
-    postfuncs = localdata.getVarFlag(task, 'postfuncs', expand=True)
+        # Dup the existing fds so we dont lose them
+        osi = [os.dup(sys.stdin.fileno()), sys.stdin.fileno()]
+        oso = [os.dup(sys.stdout.fileno()), sys.stdout.fileno()]
+        ose = [os.dup(sys.stderr.fileno()), sys.stderr.fileno()]
 
-    # Handle logfiles
-    si = file('/dev/null', 'r')
-    try:
-        logfile = file(logfn, 'w')
-    except OSError:
-        logger.exception("Opening log file '%s'", logfn)
-        pass
+        # Replace those fds with our own
+        os.dup2(si.fileno(), osi[1])
+        os.dup2(so.fileno(), oso[1])
+        os.dup2(se.fileno(), ose[1])
 
-    # Dup the existing fds so we dont lose them
-    osi = [os.dup(sys.stdin.fileno()), sys.stdin.fileno()]
-    oso = [os.dup(sys.stdout.fileno()), sys.stdout.fileno()]
-    ose = [os.dup(sys.stderr.fileno()), sys.stderr.fileno()]
+        # Since we've remapped stdout and stderr, its safe for log messages to be printed there now
+        # exec_func can nest so we have to save state
+        origstdout = bb.event.useStdout
+        bb.event.useStdout = True
 
-    # Replace those fds with our own
-    os.dup2(si.fileno(), osi[1])
-    os.dup2(logfile.fileno(), oso[1])
-    os.dup2(logfile.fileno(), ose[1])
 
-    # Ensure python logging goes to the logfile
-    handler = logging.StreamHandler(logfile)
-    handler.setFormatter(logformatter)
-    # Always enable full debug output into task logfiles
-    handler.setLevel(logging.DEBUG - 2)
-    bblogger.addHandler(handler)
-
-    localdata.setVar('BB_LOGFILE', logfn)
-
-    event.fire(TaskStarted(task, localdata), localdata)
-    try:
-        for func in (prefuncs or '').split():
+        prefuncs = (data.getVarFlag(task, 'prefuncs', localdata) or "").split()
+        for func in prefuncs:
             exec_func(func, localdata)
         exec_func(task, localdata)
-        for func in (postfuncs or '').split():
+        postfuncs = (data.getVarFlag(task, 'postfuncs', localdata) or "").split()
+        for func in postfuncs:
             exec_func(func, localdata)
-    except FuncFailed as exc:
+
+        event.fire(TaskSucceeded(task, localdata), localdata)
+
+        # make stamp, or cause event and raise exception
+        if not data.getVarFlag(task, 'nostamp', d) and not data.getVarFlag(task, 'selfstamp', d):
+            make_stamp(task, d)
+
+    except FuncFailed as message:
+        # Try to extract the optional logfile
+        try:
+            (msg, logfile) = message
+        except:
+            logfile = None
+            msg = message
         if not quieterr:
-            logger.error(str(exc))
-            event.fire(TaskFailed(task, logfn, localdata), localdata)
+            bb.msg.error(bb.msg.domain.Build, "Task failed: %s" % message )
+            failedevent = TaskFailed(msg, logfile, task, d)
+            event.fire(failedevent, d)
+        return 1
+
+    except Exception:
+        from traceback import format_exc
+        if not quieterr:
+            bb.msg.error(bb.msg.domain.Build, "Build of %s failed" % (task))
+            bb.msg.error(bb.msg.domain.Build, format_exc())
+            failedevent = TaskFailed("Task Failed", None, task, d)
+            event.fire(failedevent, d)
         return 1
     finally:
         sys.stdout.flush()
         sys.stderr.flush()
 
-        bblogger.removeHandler(handler)
+        bb.event.useStdout = origstdout
 
         # Restore the backup fds
         os.dup2(osi[0], osi[1])
         os.dup2(oso[0], oso[1])
         os.dup2(ose[0], ose[1])
 
+        # Close our logs
+        si.close()
+        so.close()
+        se.close()
+
+        if logfile and os.path.exists(logfile) and os.path.getsize(logfile) == 0:
+            bb.msg.debug(2, bb.msg.domain.Build, "Zero size logfile %s, removing" % logfile)
+            os.remove(logfile)
+            try:
+               os.remove(loglink)
+            except OSError as e:
+               pass
+
         # Close the backup fds
         os.close(osi[0])
         os.close(oso[0])
         os.close(ose[0])
-        si.close()
-
-        logfile.close()
-        if os.path.exists(logfn) and os.path.getsize(logfn) == 0:
-            logger.debug(2, "Zero size logfn %s, removing", logfn)
-            bb.utils.remove(logfn)
-            bb.utils.remove(loglink)
-    event.fire(TaskSucceeded(task, localdata), localdata)
-
-    if not localdata.getVarFlag(task, 'nostamp') and not localdata.getVarFlag(task, 'selfstamp'):
-        make_stamp(task, localdata)
 
     return 0
 
-def exec_task(fn, task, d):
-    try: 
-        quieterr = False
-        if d.getVarFlag(task, "quieterrors") is not None:
-            quieterr = True
+def extract_stamp(d, fn):
+    """
+    Extracts stamp format which is either a data dictionary (fn unset)
+    or a dataCache entry (fn set).
+    """
+    if fn:
+        return d.stamp[fn]
+    return data.getVar('STAMP', d, 1)
 
-        return _exec_task(fn, task, d, quieterr)
-    except Exception:
-        from traceback import format_exc
-        if not quieterr:
-            logger.error("Build of %s failed" % (task))
-            logger.error(format_exc())
-            failedevent = TaskFailed(task, None, d)
-            event.fire(failedevent, d)
-        return 1
-
-def stamp_internal(taskname, d, file_name):
+def stamp_internal(task, d, file_name):
     """
     Internal stamp helper function
+    Removes any stamp for the given task
     Makes sure the stamp directory exists
     Returns the stamp path+filename
-
-    In the bitbake core, d can be a CacheData and file_name will be set.
-    When called in task context, d will be a data store, file_name will not be set
     """
-    taskflagname = taskname
-    if taskname.endswith("_setscene") and taskname != "do_setscene":
-        taskflagname = taskname.replace("_setscene", "")
-
-    if file_name:
-        stamp = d.stamp_base[file_name].get(taskflagname) or d.stamp[file_name]
-        extrainfo = d.stamp_extrainfo[file_name].get(taskflagname) or ""
-    else:
-        stamp = d.getVarFlag(taskflagname, 'stamp-base', True) or d.getVar('STAMP', True)
-        file_name = d.getVar('BB_FILENAME', True)
-        extrainfo = d.getVarFlag(taskflagname, 'stamp-extra-info', True) or ""
-
+    stamp = extract_stamp(d, file_name)
     if not stamp:
         return
-
-    stamp = bb.parse.siggen.stampfile(stamp, file_name, taskname, extrainfo)
-
+    stamp = "%s.%s" % (stamp, task)
     bb.utils.mkdirhier(os.path.dirname(stamp))
-
+    # Remove the file and recreate to force timestamp
+    # change on broken NFS filesystems
+    if os.access(stamp, os.F_OK):
+        os.remove(stamp)
     return stamp
 
 def make_stamp(task, d, file_name = None):
@@ -406,33 +389,16 @@ def make_stamp(task, d, file_name = None):
     (d can be a data dict or dataCache)
     """
     stamp = stamp_internal(task, d, file_name)
-    # Remove the file and recreate to force timestamp
-    # change on broken NFS filesystems
     if stamp:
-        bb.utils.remove(stamp)
         f = open(stamp, "w")
         f.close()
 
-    # If we're in task context, write out a signature file for each task
-    # as it completes
-    if not task.endswith("_setscene") and task != "do_setscene" and not file_name:
-        file_name = d.getVar('BB_FILENAME', True)
-        bb.parse.siggen.dump_sigtask(file_name, task, d.getVar('STAMP', True), True)
-
 def del_stamp(task, d, file_name = None):
     """
     Removes a stamp for a given task
     (d can be a data dict or dataCache)
     """
-    stamp = stamp_internal(task, d, file_name)
-    bb.utils.remove(stamp)
-
-def stampfile(taskname, d, file_name = None):
-    """
-    Return the stamp for a given task
-    (d can be a data dict or dataCache)
-    """
-    return stamp_internal(taskname, d, file_name)
+    stamp_internal(task, d, file_name)
 
 def add_tasks(tasklist, d):
     task_deps = data.getVar('_task_deps', d)
@@ -463,8 +429,6 @@ def add_tasks(tasklist, d):
         getTask('recrdeptask')
         getTask('nostamp')
         getTask('fakeroot')
-        getTask('noexec')
-        getTask('umask')
         task_deps['parents'][task] = []
         for dep in flags['deps']:
             dep = data.expand(dep, d)

bitbake/lib/bb/cache.py

@@ -29,237 +29,27 @@
 
 
 import os
-import logging
-from collections import defaultdict
 import bb.data
 import bb.utils
 
-logger = logging.getLogger("BitBake.Cache")
-
 try:
     import cPickle as pickle
 except ImportError:
     import pickle
-    logger.info("Importing cPickle failed. "
-                "Falling back to a very slow implementation.")
+    bb.msg.note(1, bb.msg.domain.Cache, "Importing cPickle failed. Falling back to a very slow implementation.")
 
-__cache_version__ = "142"
+__cache_version__ = "132"
 
-def getCacheFile(path, filename):
-    return os.path.join(path, filename)
-
-# RecipeInfoCommon defines common data retrieving methods
-# from meta data for caches. CoreRecipeInfo as well as other
-# Extra RecipeInfo needs to inherit this class
-class RecipeInfoCommon(object):
-
-    @classmethod
-    def listvar(cls, var, metadata):
-        return cls.getvar(var, metadata).split()
-
-    @classmethod
-    def intvar(cls, var, metadata):
-        return int(cls.getvar(var, metadata) or 0)
-
-    @classmethod
-    def depvar(cls, var, metadata):
-        return bb.utils.explode_deps(cls.getvar(var, metadata))
-
-    @classmethod
-    def pkgvar(cls, var, packages, metadata):
-        return dict((pkg, cls.depvar("%s_%s" % (var, pkg), metadata))
-                    for pkg in packages)
-
-    @classmethod
-    def taskvar(cls, var, tasks, metadata):
-        return dict((task, cls.getvar("%s_task-%s" % (var, task), metadata))
-                    for task in tasks)
-
-    @classmethod
-    def flaglist(cls, flag, varlist, metadata):
-        return dict((var, metadata.getVarFlag(var, flag, True))
-                    for var in varlist)
-
-    @classmethod
-    def getvar(cls, var, metadata):
-        return metadata.getVar(var, True) or ''
-
-
-class CoreRecipeInfo(RecipeInfoCommon):
-    __slots__ = ()
-
-    cachefile = "bb_cache.dat"   
-
-    def __init__(self, filename, metadata):      
-        self.file_depends = metadata.getVar('__depends', False)
-        self.timestamp = bb.parse.cached_mtime(filename)
-        self.variants = self.listvar('__VARIANTS', metadata) + ['']
-        self.appends = self.listvar('__BBAPPEND', metadata)
-        self.nocache = self.getvar('__BB_DONT_CACHE', metadata)
-
-        self.skipreason = self.getvar('__SKIPPED', metadata)
-        if self.skipreason:
-            self.pn = self.getvar('PN', metadata) or bb.parse.BBHandler.vars_from_file(filename,metadata)[0]
-            self.skipped = True
-            self.provides  = self.depvar('PROVIDES', metadata)
-            self.rprovides = self.depvar('RPROVIDES', metadata)
-            return
-
-        self.tasks = metadata.getVar('__BBTASKS', False)
-
-        self.pn = self.getvar('PN', metadata)
-        self.packages = self.listvar('PACKAGES', metadata)
-        if not self.pn in self.packages:
-            self.packages.append(self.pn)
-
-        self.basetaskhashes = self.taskvar('BB_BASEHASH', self.tasks, metadata)
-        self.hashfilename = self.getvar('BB_HASHFILENAME', metadata)
-
-        self.file_depends = metadata.getVar('__depends', False)
-        self.task_deps = metadata.getVar('_task_deps', False) or {'tasks': [], 'parents': {}}
-
-        self.skipped = False
-        self.pe = self.getvar('PE', metadata)
-        self.pv = self.getvar('PV', metadata)
-        self.pr = self.getvar('PR', metadata)
-        self.defaultpref = self.intvar('DEFAULT_PREFERENCE', metadata)
-        self.broken = self.getvar('BROKEN', metadata)
-        self.not_world = self.getvar('EXCLUDE_FROM_WORLD', metadata)
-        self.stamp = self.getvar('STAMP', metadata)
-        self.stamp_base = self.flaglist('stamp-base', self.tasks, metadata)
-        self.stamp_extrainfo = self.flaglist('stamp-extra-info', self.tasks, metadata)
-        self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC', metadata)
-        self.depends          = self.depvar('DEPENDS', metadata)
-        self.provides         = self.depvar('PROVIDES', metadata)
-        self.rdepends         = self.depvar('RDEPENDS', metadata)
-        self.rprovides        = self.depvar('RPROVIDES', metadata)
-        self.rrecommends      = self.depvar('RRECOMMENDS', metadata)
-        self.rprovides_pkg    = self.pkgvar('RPROVIDES', self.packages, metadata)
-        self.rdepends_pkg     = self.pkgvar('RDEPENDS', self.packages, metadata)
-        self.rrecommends_pkg  = self.pkgvar('RRECOMMENDS', self.packages, metadata)
-        self.inherits         = self.getvar('__inherit_cache', metadata)
-        self.summary          = self.getvar('SUMMARY', metadata)
-        self.license          = self.getvar('LICENSE', metadata)
-        self.section          = self.getvar('SECTION', metadata)
-        self.fakerootenv      = self.getvar('FAKEROOTENV', metadata)
-        self.fakerootdirs     = self.getvar('FAKEROOTDIRS', metadata)
-
-    @classmethod
-    def init_cacheData(cls, cachedata):
-        # CacheData in Core RecipeInfo Class
-        cachedata.task_deps = {}
-        cachedata.pkg_fn = {}
-        cachedata.pkg_pn = defaultdict(list)
-        cachedata.pkg_pepvpr = {}
-        cachedata.pkg_dp = {}
-
-        cachedata.stamp = {}
-        cachedata.stamp_base = {}
-        cachedata.stamp_extrainfo = {}
-        cachedata.fn_provides = {}
-        cachedata.pn_provides = defaultdict(list)
-        cachedata.all_depends = []
-
-        cachedata.deps = defaultdict(list)
-        cachedata.packages = defaultdict(list)
-        cachedata.providers = defaultdict(list)
-        cachedata.rproviders = defaultdict(list)
-        cachedata.packages_dynamic = defaultdict(list)
-
-        cachedata.rundeps = defaultdict(lambda: defaultdict(list))
-        cachedata.runrecs = defaultdict(lambda: defaultdict(list))
-        cachedata.possible_world = []
-        cachedata.universe_target = []
-        cachedata.hashfn = {}
-
-        cachedata.basetaskhash = {}
-        cachedata.inherits = {}
-        cachedata.summary = {}
-        cachedata.license = {}
-        cachedata.section = {}
-        cachedata.fakerootenv = {}
-        cachedata.fakerootdirs = {}
-
-    def add_cacheData(self, cachedata, fn):
-        cachedata.task_deps[fn] = self.task_deps
-        cachedata.pkg_fn[fn] = self.pn
-        cachedata.pkg_pn[self.pn].append(fn)
-        cachedata.pkg_pepvpr[fn] = (self.pe, self.pv, self.pr)
-        cachedata.pkg_dp[fn] = self.defaultpref
-        cachedata.stamp[fn] = self.stamp
-        cachedata.stamp_base[fn] = self.stamp_base
-        cachedata.stamp_extrainfo[fn] = self.stamp_extrainfo
-
-        provides = [self.pn]
-        for provide in self.provides:
-            if provide not in provides:
-                provides.append(provide)
-        cachedata.fn_provides[fn] = provides
-
-        for provide in provides:
-            cachedata.providers[provide].append(fn)
-            if provide not in cachedata.pn_provides[self.pn]:
-                cachedata.pn_provides[self.pn].append(provide)
-
-        for dep in self.depends:
-            if dep not in cachedata.deps[fn]:
-                cachedata.deps[fn].append(dep)
-            if dep not in cachedata.all_depends:
-                cachedata.all_depends.append(dep)
-
-        rprovides = self.rprovides
-        for package in self.packages:
-            cachedata.packages[package].append(fn)
-            rprovides += self.rprovides_pkg[package]
-
-        for rprovide in rprovides:
-            cachedata.rproviders[rprovide].append(fn)
-
-        for package in self.packages_dynamic:
-            cachedata.packages_dynamic[package].append(fn)
-
-        # Build hash of runtime depends and rececommends
-        for package in self.packages + [self.pn]:
-            cachedata.rundeps[fn][package] = list(self.rdepends) + self.rdepends_pkg[package]
-            cachedata.runrecs[fn][package] = list(self.rrecommends) + self.rrecommends_pkg[package]
-
-        # Collect files we may need for possible world-dep
-        # calculations
-        if not self.broken and not self.not_world:
-            cachedata.possible_world.append(fn)
-
-        # create a collection of all targets for sanity checking
-        # tasks, such as upstream versions, license, and tools for
-        # task and image creation.
-        cachedata.universe_target.append(self.pn)
-
-        cachedata.hashfn[fn] = self.hashfilename
-        for task, taskhash in self.basetaskhashes.iteritems():
-            identifier = '%s.%s' % (fn, task)
-            cachedata.basetaskhash[identifier] = taskhash
-
-        cachedata.inherits[fn] = self.inherits
-        cachedata.summary[fn] = self.summary
-        cachedata.license[fn] = self.license
-        cachedata.section[fn] = self.section
-        cachedata.fakerootenv[fn] = self.fakerootenv
-        cachedata.fakerootdirs[fn] = self.fakerootdirs
-
-
-
-class Cache(object):
+class Cache:
     """
     BitBake Cache implementation
     """
+    def __init__(self, data):
+
 
-    def __init__(self, data, caches_array):
-        # Pass caches_array information into Cache Constructor
-        # It will be used in later for deciding whether we 
-        # need extra cache file dump/load support 
-        self.caches_array = caches_array
         self.cachedir = bb.data.getVar("CACHE", data, True)
-        self.clean = set()
-        self.checked = set()
+        self.clean = {}
+        self.checked = {}
         self.depends_cache = {}
         self.data = None
         self.data_fn = None
@@ -267,104 +57,92 @@ class Cache(object):
 
         if self.cachedir in [None, '']:
             self.has_cache = False
-            logger.info("Not using a cache. "
-                        "Set CACHE = <directory> to enable.")
+            bb.msg.note(1, bb.msg.domain.Cache, "Not using a cache. Set CACHE = <directory> to enable.")
             return
 
         self.has_cache = True
-        self.cachefile = getCacheFile(self.cachedir, "bb_cache.dat")
+        self.cachefile = os.path.join(self.cachedir, "bb_cache.dat")
 
-        logger.debug(1, "Using cache in '%s'", self.cachedir)
+        bb.msg.debug(1, bb.msg.domain.Cache, "Using cache in '%s'" % self.cachedir)
         bb.utils.mkdirhier(self.cachedir)
 
         # If any of configuration.data's dependencies are newer than the
         # cache there isn't even any point in loading it...
         newest_mtime = 0
-        deps = bb.data.getVar("__base_depends", data)
+        deps = bb.data.getVar("__depends", data)
 
-        old_mtimes = [old_mtime for _, old_mtime in deps]
+        old_mtimes = [old_mtime for f, old_mtime in deps]
         old_mtimes.append(newest_mtime)
         newest_mtime = max(old_mtimes)
 
-        cache_ok = True
-        if self.caches_array:
-            for cache_class in self.caches_array:
-                if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
-                    cachefile = getCacheFile(self.cachedir, cache_class.cachefile)
-                    cache_ok = cache_ok and (bb.parse.cached_mtime_noerror(cachefile) >= newest_mtime)
-                    cache_class.init_cacheData(self)
-        if cache_ok:
-            self.load_cachefile()
-        elif os.path.isfile(self.cachefile):
-            logger.info("Out of date cache found, rebuilding...")
-
-    def load_cachefile(self):
-        # Firstly, using core cache file information for
-        # valid checking
-        with open(self.cachefile, "rb") as cachefile:
-            pickled = pickle.Unpickler(cachefile)
+        if bb.parse.cached_mtime_noerror(self.cachefile) >= newest_mtime:
             try:
-                cache_ver = pickled.load()
-                bitbake_ver = pickled.load()
-            except Exception:
-                logger.info('Invalid cache, rebuilding...')
-                return
+                p = pickle.Unpickler(file(self.cachefile, "rb"))
+                self.depends_cache, version_data = p.load()
+                if version_data['CACHE_VER'] != __cache_version__:
+                    raise ValueError('Cache Version Mismatch')
+                if version_data['BITBAKE_VER'] != bb.__version__:
+                    raise ValueError('Bitbake Version Mismatch')
+            except EOFError:
+                bb.msg.note(1, bb.msg.domain.Cache, "Truncated cache found, rebuilding...")
+                self.depends_cache = {}
+            except:
+                bb.msg.note(1, bb.msg.domain.Cache, "Invalid cache found, rebuilding...")
+                self.depends_cache = {}
+        else:
+            if os.path.isfile(self.cachefile):
+                bb.msg.note(1, bb.msg.domain.Cache, "Out of date cache found, rebuilding...")
 
-            if cache_ver != __cache_version__:
-                logger.info('Cache version mismatch, rebuilding...')
-                return
-            elif bitbake_ver != bb.__version__:
-                logger.info('Bitbake version mismatch, rebuilding...')
-                return
+    def getVar(self, var, fn, exp = 0):
+        """
+        Gets the value of a variable
+        (similar to getVar in the data class)
 
+        There are two scenarios:
+          1. We have cached data - serve from depends_cache[fn]
+          2. We're learning what data to cache - serve from data
+             backend but add a copy of the data to the cache.
+        """
+        if fn in self.clean:
+            return self.depends_cache[fn][var]
 
-        cachesize = 0
-        previous_progress = 0
-        previous_percent = 0
+        self.depends_cache.setdefault(fn, {})
 
-        # Calculate the correct cachesize of all those cache files
-        for cache_class in self.caches_array:
-            if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
-                cachefile = getCacheFile(self.cachedir, cache_class.cachefile)
-                with open(cachefile, "rb") as cachefile:
-                    cachesize += os.fstat(cachefile.fileno()).st_size
+        if fn != self.data_fn:
+            # We're trying to access data in the cache which doesn't exist
+            # yet setData hasn't been called to setup the right access. Very bad.
+            bb.msg.error(bb.msg.domain.Cache, "Parsing error data_fn %s and fn %s don't match" % (self.data_fn, fn))
 
-        bb.event.fire(bb.event.CacheLoadStarted(cachesize), self.data)
-        
-        for cache_class in self.caches_array:
-            if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
-                cachefile = getCacheFile(self.cachedir, cache_class.cachefile)
-                with open(cachefile, "rb") as cachefile:
-                    pickled = pickle.Unpickler(cachefile)                    
-                    while cachefile:
-                        try:
-                            key = pickled.load()
-                            value = pickled.load()
-                        except Exception:
-                            break
-                        if self.depends_cache.has_key(key):
-                            self.depends_cache[key].append(value)
-                        else:
-                            self.depends_cache[key] = [value]
-                        # only fire events on even percentage boundaries
-                        current_progress = cachefile.tell() + previous_progress
-                        current_percent = 100 * current_progress / cachesize
-                        if current_percent > previous_percent:
-                            previous_percent = current_percent
-                            bb.event.fire(bb.event.CacheLoadProgress(current_progress),
-                                          self.data)
+        self.cacheclean = False
+        result = bb.data.getVar(var, self.data, exp)
+        self.depends_cache[fn][var] = result
+        return result
 
-                    previous_progress += current_progress
+    def setData(self, virtualfn, fn, data):
+        """
+        Called to prime bb_cache ready to learn which variables to cache.
+        Will be followed by calls to self.getVar which aren't cached
+        but can be fulfilled from self.data.
+        """
+        self.data_fn = virtualfn
+        self.data = data
 
-        # Note: depends cache number is corresponding to the parsing file numbers.
-        # The same file has several caches, still regarded as one item in the cache
-        bb.event.fire(bb.event.CacheLoadCompleted(cachesize,
-                                                  len(self.depends_cache)),
-                      self.data)
+        # Make sure __depends makes the depends_cache
+        # If we're a virtual class we need to make sure all our depends are appended
+        # to the depends of fn.
+        depends = self.getVar("__depends", virtualfn) or set()
+        self.depends_cache.setdefault(fn, {})
+        if "__depends" not in self.depends_cache[fn] or not self.depends_cache[fn]["__depends"]:
+            self.depends_cache[fn]["__depends"] = depends
+        else:
+            self.depends_cache[fn]["__depends"].update(depends)
 
-    
-    @staticmethod
-    def virtualfn2realfn(virtualfn):
+        # Make sure the variants always make it into the cache too
+        self.getVar('__VARIANTS', virtualfn, True)
+
+        self.depends_cache[virtualfn]["CACHETIMESTAMP"] = bb.parse.cached_mtime(fn)
+
+    def virtualfn2realfn(self, virtualfn):
         """
         Convert a virtual file name to a real one + the associated subclass keyword
         """
@@ -372,105 +150,81 @@ class Cache(object):
         fn = virtualfn
         cls = ""
         if virtualfn.startswith('virtual:'):
-            elems = virtualfn.split(':')
-            cls = ":".join(elems[1:-1])
-            fn = elems[-1]
+            cls = virtualfn.split(':', 2)[1]
+            fn = virtualfn.replace('virtual:' + cls + ':', '')
+        #bb.msg.debug(2, bb.msg.domain.Cache, "virtualfn2realfn %s to %s %s" % (virtualfn, fn, cls))
         return (fn, cls)
 
-    @staticmethod
-    def realfn2virtual(realfn, cls):
+    def realfn2virtual(self, realfn, cls):
         """
         Convert a real filename + the associated subclass keyword to a virtual filename
         """
         if cls == "":
+            #bb.msg.debug(2, bb.msg.domain.Cache, "realfn2virtual %s and '%s' to %s" % (realfn, cls, realfn))
             return realfn
+        #bb.msg.debug(2, bb.msg.domain.Cache, "realfn2virtual %s and %s to %s" % (realfn, cls, "virtual:" + cls + ":" + realfn))
         return "virtual:" + cls + ":" + realfn
 
-    @classmethod
-    def loadDataFull(cls, virtualfn, appends, cfgData):
+    def loadDataFull(self, virtualfn, appends, cfgData):
         """
         Return a complete set of data for fn.
         To do this, we need to parse the file.
         """
 
-        (fn, virtual) = cls.virtualfn2realfn(virtualfn)
+        (fn, cls) = self.virtualfn2realfn(virtualfn)
 
-        logger.debug(1, "Parsing %s (full)", fn)
+        bb.msg.debug(1, bb.msg.domain.Cache, "Parsing %s (full)" % fn)
 
-        cfgData.setVar("__ONLYFINALISE", virtual or "default")
-        bb_data = cls.load_bbfile(fn, appends, cfgData)
-        return bb_data[virtual]
-
-    @classmethod
-    def parse(cls, filename, appends, configdata, caches_array):
-        """Parse the specified filename, returning the recipe information"""
-        infos = []
-        datastores = cls.load_bbfile(filename, appends, configdata)
-        depends = set()
-        for variant, data in sorted(datastores.iteritems(),
-                                    key=lambda i: i[0],
-                                    reverse=True):
-            virtualfn = cls.realfn2virtual(filename, variant)
-            depends |= (data.getVar("__depends", False) or set())
-            if depends and not variant:
-                data.setVar("__depends", depends)
-
-            info_array = []
-            for cache_class in caches_array:
-                if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
-                    info = cache_class(filename, data)
-                    info_array.append(info)
-            infos.append((virtualfn, info_array))
-
-        return infos
-
-    def load(self, filename, appends, configdata):
-        """Obtain the recipe information for the specified filename,
-        using cached values if available, otherwise parsing.
-
-        Note that if it does parse to obtain the info, it will not
-        automatically add the information to the cache or to your
-        CacheData.  Use the add or add_info method to do so after
-        running this, or use loadData instead."""
-        cached = self.cacheValid(filename, appends)
-        if cached:
-            infos = []
-            # info_array item is a list of [CoreRecipeInfo, XXXRecipeInfo]
-            info_array = self.depends_cache[filename]
-            for variant in info_array[0].variants:
-                virtualfn = self.realfn2virtual(filename, variant)
-                infos.append((virtualfn, self.depends_cache[virtualfn]))
-        else:
-            logger.debug(1, "Parsing %s", filename)
-            return self.parse(filename, appends, configdata, self.caches_array)
-
-        return cached, infos
+        bb_data = self.load_bbfile(fn, appends, cfgData)
+        return bb_data[cls]
 
     def loadData(self, fn, appends, cfgData, cacheData):
-        """Load the recipe info for the specified filename,
-        parsing and adding to the cache if necessary, and adding
-        the recipe information to the supplied CacheData instance."""
-        skipped, virtuals = 0, 0
+        """
+        Load a subset of data for fn.
+        If the cached data is valid we do nothing,
+        To do this, we need to parse the file and set the system
+        to record the variables accessed.
+        Return the cache status and whether the file was skipped when parsed
+        """
+        skipped = 0
+        virtuals = 0
 
-        cached, infos = self.load(fn, appends, cfgData)
-        for virtualfn, info_array in infos:
-            if info_array[0].skipped:
-                logger.debug(1, "Skipping %s: %s", virtualfn, info_array[0].skipreason)
-                skipped += 1
-            else:
-                self.add_info(virtualfn, info_array, cacheData, not cached)
+        if fn not in self.checked:
+            self.cacheValidUpdate(fn)
+
+        if self.cacheValid(fn):
+            multi = self.getVar('__VARIANTS', fn, True)
+            for cls in (multi or "").split() + [""]:
+                virtualfn = self.realfn2virtual(fn, cls)
+                if self.depends_cache[virtualfn]["__SKIPPED"]:
+                    skipped += 1
+                    bb.msg.debug(1, bb.msg.domain.Cache, "Skipping %s" % virtualfn)
+                    continue
+                self.handle_data(virtualfn, cacheData)
                 virtuals += 1
+            return True, skipped, virtuals
 
-        return cached, skipped, virtuals
+        bb.msg.debug(1, bb.msg.domain.Cache, "Parsing %s" % fn)
 
-    def cacheValid(self, fn, appends):
+        bb_data = self.load_bbfile(fn, appends, cfgData)
+
+        for data in bb_data:
+            virtualfn = self.realfn2virtual(fn, data)
+            self.setData(virtualfn, fn, bb_data[data])
+            if self.getVar("__SKIPPED", virtualfn):
+                skipped += 1
+                bb.msg.debug(1, bb.msg.domain.Cache, "Skipping %s" % virtualfn)
+            else:
+                self.handle_data(virtualfn, cacheData)
+                virtuals += 1
+        return False, skipped, virtuals
+
+
+    def cacheValid(self, fn):
         """
         Is the cache valid for fn?
         Fast version, no timestamps checked.
         """
-        if fn not in self.checked:
-            self.cacheValidUpdate(fn, appends)
-
         # Is cache enabled?
         if not self.has_cache:
             return False
@@ -478,7 +232,7 @@ class Cache(object):
             return True
         return False
 
-    def cacheValidUpdate(self, fn, appends):
+    def cacheValidUpdate(self, fn):
         """
         Is the cache valid for fn?
         Make thorough (slower) checks including timestamps.
@@ -487,73 +241,70 @@ class Cache(object):
         if not self.has_cache:
             return False
 
-        self.checked.add(fn)
+        self.checked[fn] = ""
+
+        # Pretend we're clean so getVar works
+        self.clean[fn] = ""
 
         # File isn't in depends_cache
         if not fn in self.depends_cache:
-            logger.debug(2, "Cache: %s is not cached", fn)
+            bb.msg.debug(2, bb.msg.domain.Cache, "Cache: %s is not cached" % fn)
+            self.remove(fn)
             return False
 
         mtime = bb.parse.cached_mtime_noerror(fn)
 
         # Check file still exists
         if mtime == 0:
-            logger.debug(2, "Cache: %s no longer exists", fn)
+            bb.msg.debug(2, bb.msg.domain.Cache, "Cache: %s no longer exists" % fn)
             self.remove(fn)
             return False
 
-        info_array = self.depends_cache[fn]
         # Check the file's timestamp
-        if mtime != info_array[0].timestamp:
-            logger.debug(2, "Cache: %s changed", fn)
+        if mtime != self.getVar("CACHETIMESTAMP", fn, True):
+            bb.msg.debug(2, bb.msg.domain.Cache, "Cache: %s changed" % fn)
             self.remove(fn)
             return False
 
         # Check dependencies are still valid
-        depends = info_array[0].file_depends
+        depends = self.getVar("__depends", fn, True)
         if depends:
             for f, old_mtime in depends:
                 fmtime = bb.parse.cached_mtime_noerror(f)
                 # Check if file still exists
                 if old_mtime != 0 and fmtime == 0:
-                    logger.debug(2, "Cache: %s's dependency %s was removed",
-                                    fn, f)
                     self.remove(fn)
                     return False
 
                 if (fmtime != old_mtime):
-                    logger.debug(2, "Cache: %s's dependency %s changed",
-                                    fn, f)
+                    bb.msg.debug(2, bb.msg.domain.Cache, "Cache: %s's dependency %s changed" % (fn, f))
                     self.remove(fn)
                     return False
 
-        if appends != info_array[0].appends:
-            logger.debug(2, "Cache: appends for %s changed", fn)
-            bb.note("%s to %s" % (str(appends), str(info_array[0].appends)))
-            self.remove(fn)
-            return False
+        #bb.msg.debug(2, bb.msg.domain.Cache, "Depends Cache: %s is clean" % fn)
+        if not fn in self.clean:
+            self.clean[fn] = ""
 
         invalid = False
-        for cls in info_array[0].variants:
+        # Mark extended class data as clean too
+        multi = self.getVar('__VARIANTS', fn, True)
+        for cls in (multi or "").split():
             virtualfn = self.realfn2virtual(fn, cls)
-            self.clean.add(virtualfn)
-            if virtualfn not in self.depends_cache:
-                logger.debug(2, "Cache: %s is not cached", virtualfn)
+            self.clean[virtualfn] = ""
+            if not virtualfn in self.depends_cache:
+                bb.msg.debug(2, bb.msg.domain.Cache, "Cache: %s is not cached" % virtualfn)
                 invalid = True
 
-        # If any one of the variants is not present, mark as invalid for all
+        # If any one of the varients is not present, mark cache as invalid for all
         if invalid:
-            for cls in info_array[0].variants:
+            for cls in (multi or "").split():
                 virtualfn = self.realfn2virtual(fn, cls)
-                if virtualfn in self.clean:
-                    logger.debug(2, "Cache: Removing %s from cache", virtualfn)
-                    self.clean.remove(virtualfn)
-            if fn in self.clean:
-                logger.debug(2, "Cache: Marking %s as not clean", fn)
-                self.clean.remove(fn)
+                bb.msg.debug(2, bb.msg.domain.Cache, "Cache: Removing %s from cache" % virtualfn)
+                del self.clean[virtualfn]
+            bb.msg.debug(2, bb.msg.domain.Cache, "Cache: Removing %s from cache" % fn)
+            del self.clean[fn]
             return False
 
-        self.clean.add(fn)
         return True
 
     def remove(self, fn):
@@ -561,84 +312,154 @@ class Cache(object):
         Remove a fn from the cache
         Called from the parser in error cases
         """
+        bb.msg.debug(1, bb.msg.domain.Cache, "Removing %s from cache" % fn)
         if fn in self.depends_cache:
-            logger.debug(1, "Removing %s from cache", fn)
             del self.depends_cache[fn]
         if fn in self.clean:
-            logger.debug(1, "Marking %s as unclean", fn)
-            self.clean.remove(fn)
+            del self.clean[fn]
 
     def sync(self):
         """
         Save the cache
         Called from the parser when complete (or exiting)
         """
+        import copy
 
         if not self.has_cache:
             return
 
         if self.cacheclean:
-            logger.debug(2, "Cache is clean, not saving.")
+            bb.msg.note(1, bb.msg.domain.Cache, "Cache is clean, not saving.")
             return
 
-        file_dict = {}
-        pickler_dict = {}
-        for cache_class in self.caches_array:
-            if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
-                cache_class_name = cache_class.__name__
-                cachefile = getCacheFile(self.cachedir, cache_class.cachefile)
-                file_dict[cache_class_name] = open(cachefile, "wb")
-                pickler_dict[cache_class_name] =  pickle.Pickler(file_dict[cache_class_name], pickle.HIGHEST_PROTOCOL)
-                   
-        pickler_dict['CoreRecipeInfo'].dump(__cache_version__)
-        pickler_dict['CoreRecipeInfo'].dump(bb.__version__)
+        version_data = {}
+        version_data['CACHE_VER'] = __cache_version__
+        version_data['BITBAKE_VER'] = bb.__version__
 
-        try:
-            for key, info_array in self.depends_cache.iteritems():
-                for info in info_array:
-                    if isinstance(info, RecipeInfoCommon):
-                        cache_class_name = info.__class__.__name__
-                        pickler_dict[cache_class_name].dump(key)
-                        pickler_dict[cache_class_name].dump(info)
-        finally:
-            for cache_class in self.caches_array:
-                if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
-                    cache_class_name = cache_class.__name__
-                    file_dict[cache_class_name].close()
+        cache_data = copy.copy(self.depends_cache)
+        for fn in self.depends_cache:
+            if '__BB_DONT_CACHE' in self.depends_cache[fn] and self.depends_cache[fn]['__BB_DONT_CACHE']:
+                bb.msg.debug(2, bb.msg.domain.Cache, "Not caching %s, marked as not cacheable" % fn)
+                del cache_data[fn]
+            elif 'PV' in self.depends_cache[fn] and 'SRCREVINACTION' in self.depends_cache[fn]['PV']:
+                bb.msg.error(bb.msg.domain.Cache, "Not caching %s as it had SRCREVINACTION in PV. Please report this bug" % fn)
+                del cache_data[fn]
 
-        del self.depends_cache
+        p = pickle.Pickler(file(self.cachefile, "wb" ), -1 )
+        p.dump([cache_data, version_data])
 
-    @staticmethod
-    def mtime(cachefile):
+    def mtime(self, cachefile):
         return bb.parse.cached_mtime_noerror(cachefile)
 
-    def add_info(self, filename, info_array, cacheData, parsed=None):
-        if isinstance(info_array[0], CoreRecipeInfo) and (not info_array[0].skipped):
-            cacheData.add_from_recipeinfo(filename, info_array)
-
-        if not self.has_cache:
-            return
-
-        if (info_array[0].skipped or 'SRCREVINACTION' not in info_array[0].pv) and not info_array[0].nocache:
-            if parsed:
-                self.cacheclean = False
-            self.depends_cache[filename] = info_array
-
-    def add(self, file_name, data, cacheData, parsed=None):
+    def handle_data(self, file_name, cacheData):
         """
         Save data we need into the cache
         """
 
-        realfn = self.virtualfn2realfn(file_name)[0]
+        pn       = self.getVar('PN', file_name, True)
+        pe       = self.getVar('PE', file_name, True) or "0"
+        pv       = self.getVar('PV', file_name, True)
+        if 'SRCREVINACTION' in pv:
+            bb.msg.note(1, bb.msg.domain.Cache, "Found SRCREVINACTION in PV (%s) or %s. Please report this bug." % (pv, file_name))
+        pr       = self.getVar('PR', file_name, True)
+        dp       = int(self.getVar('DEFAULT_PREFERENCE', file_name, True) or "0")
+        depends   = bb.utils.explode_deps(self.getVar("DEPENDS", file_name, True) or "")
+        packages  = (self.getVar('PACKAGES', file_name, True) or "").split()
+        packages_dynamic = (self.getVar('PACKAGES_DYNAMIC', file_name, True) or "").split()
+        rprovides = (self.getVar("RPROVIDES", file_name, True) or "").split()
 
-        info_array = []
-        for cache_class in self.caches_array:
-            if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
-                info_array.append(cache_class(realfn, data))
-        self.add_info(file_name, info_array, cacheData, parsed)
+        cacheData.task_deps[file_name] = self.getVar("_task_deps", file_name)
 
-    @staticmethod
-    def load_bbfile(bbfile, appends, config):
+        # build PackageName to FileName lookup table
+        if pn not in cacheData.pkg_pn:
+            cacheData.pkg_pn[pn] = []
+        cacheData.pkg_pn[pn].append(file_name)
+
+        cacheData.stamp[file_name] = self.getVar('STAMP', file_name, True)
+
+        cacheData.tasks[file_name] = self.getVar('__BBTASKS', file_name, True)
+        for t in cacheData.tasks[file_name]:
+            cacheData.basetaskhash[file_name + "." + t] = self.getVar("BB_BASEHASH_task-%s" % t, file_name, True)
+
+        # build FileName to PackageName lookup table
+        cacheData.pkg_fn[file_name] = pn
+        cacheData.pkg_pepvpr[file_name] = (pe, pv, pr)
+        cacheData.pkg_dp[file_name] = dp
+
+        provides = [pn]
+        for provide in (self.getVar("PROVIDES", file_name, True) or "").split():
+            if provide not in provides:
+                provides.append(provide)
+
+        # Build forward and reverse provider hashes
+        # Forward: virtual -> [filenames]
+        # Reverse: PN -> [virtuals]
+        if pn not in cacheData.pn_provides:
+            cacheData.pn_provides[pn] = []
+
+        cacheData.fn_provides[file_name] = provides
+        for provide in provides:
+            if provide not in cacheData.providers:
+                cacheData.providers[provide] = []
+            cacheData.providers[provide].append(file_name)
+            if not provide in cacheData.pn_provides[pn]:
+                cacheData.pn_provides[pn].append(provide)
+
+        cacheData.deps[file_name] = []
+        for dep in depends:
+            if not dep in cacheData.deps[file_name]:
+                cacheData.deps[file_name].append(dep)
+            if not dep in cacheData.all_depends:
+                cacheData.all_depends.append(dep)
+
+        # Build reverse hash for PACKAGES, so runtime dependencies
+        # can be be resolved (RDEPENDS, RRECOMMENDS etc.)
+        for package in packages:
+            if not package in cacheData.packages:
+                cacheData.packages[package] = []
+            cacheData.packages[package].append(file_name)
+            rprovides += (self.getVar("RPROVIDES_%s" % package, file_name, 1) or "").split()
+
+        for package in packages_dynamic:
+            if not package in cacheData.packages_dynamic:
+                cacheData.packages_dynamic[package] = []
+            cacheData.packages_dynamic[package].append(file_name)
+
+        for rprovide in rprovides:
+            if not rprovide in cacheData.rproviders:
+                cacheData.rproviders[rprovide] = []
+            cacheData.rproviders[rprovide].append(file_name)
+
+        # Build hash of runtime depends and rececommends
+
+        if not file_name in cacheData.rundeps:
+            cacheData.rundeps[file_name] = {}
+        if not file_name in cacheData.runrecs:
+            cacheData.runrecs[file_name] = {}
+
+        rdepends = self.getVar('RDEPENDS', file_name, True) or ""
+        rrecommends = self.getVar('RRECOMMENDS', file_name, True) or ""
+        for package in packages + [pn]:
+            if not package in cacheData.rundeps[file_name]:
+                cacheData.rundeps[file_name][package] = []
+            if not package in cacheData.runrecs[file_name]:
+                cacheData.runrecs[file_name][package] = []
+
+            cacheData.rundeps[file_name][package] = rdepends + " " + (self.getVar("RDEPENDS_%s" % package, file_name, True) or "")
+            cacheData.runrecs[file_name][package] = rrecommends + " " + (self.getVar("RRECOMMENDS_%s" % package, file_name, True) or "")
+
+        # Collect files we may need for possible world-dep
+        # calculations
+        if not self.getVar('BROKEN', file_name, True) and not self.getVar('EXCLUDE_FROM_WORLD', file_name, True):
+            cacheData.possible_world.append(file_name)
+
+        cacheData.hashfn[file_name] = self.getVar('BB_HASHFILENAME', file_name, True)
+
+        # Touch this to make sure its in the cache
+        self.getVar('__BB_DONT_CACHE', file_name, True)
+        self.getVar('__VARIANTS', file_name, True)
+
+    def load_bbfile(self, bbfile, appends, config):
         """
         Load and parse one .bb build file
         Return the data and whether parsing resulted in the file being skipped
@@ -664,16 +485,13 @@ class Cache(object):
         try:
             if appends:
                 data.setVar('__BBAPPEND', " ".join(appends), bb_data)
-            bb_data = parse.handle(bbfile, bb_data)
-            if chdir_back:
-                os.chdir(oldpath)
+            bb_data = parse.handle(bbfile, bb_data) # read .bb data
+            if chdir_back: os.chdir(oldpath)
             return bb_data
         except:
-            if chdir_back:
-                os.chdir(oldpath)
+            if chdir_back: os.chdir(oldpath)
             raise
 
-
 def init(cooker):
     """
     The Objective: Cache the minimum amount of data possible yet get to the
@@ -694,28 +512,48 @@ def init(cooker):
     return Cache(cooker.configuration.data)
 
 
-class CacheData(object):
+
+#============================================================================#
+# CacheData
+#============================================================================#
+class CacheData:
     """
     The data structures we compile from the cached data
     """
 
-    def __init__(self, caches_array):
-        self.caches_array = caches_array
-        for cache_class in self.caches_array:
-            if type(cache_class) is type and issubclass(cache_class, RecipeInfoCommon):
-                cache_class.init_cacheData(self)        
-
-        # Direct cache variables
+    def __init__(self):
+        """
+        Direct cache variables
+        (from Cache.handle_data)
+        """
+        self.providers   = {}
+        self.rproviders = {}
+        self.packages = {}
+        self.packages_dynamic = {}
+        self.possible_world = []
+        self.pkg_pn = {}
+        self.pkg_fn = {}
+        self.pkg_pepvpr = {}
+        self.pkg_dp = {}
+        self.pn_provides = {}
+        self.fn_provides = {}
+        self.all_depends = []
+        self.deps = {}
+        self.rundeps = {}
+        self.runrecs = {}
         self.task_queues = {}
+        self.task_deps = {}
+        self.stamp = {}
         self.preferred = {}
         self.tasks = {}
-        # Indirect Cache variables (set elsewhere)
+        self.basetaskhash = {}
+        self.hashfn = {}
+
+        """
+        Indirect Cache variables
+        (set elsewhere)
+        """
         self.ignored_dependencies = []
         self.world_target = set()
         self.bbfile_priority = {}
-
-    def add_from_recipeinfo(self, fn, info_array):
-        for info in info_array:
-            info.add_cacheData(self, fn)
-
-        
+        self.bbfile_config_priorities = []

bitbake/lib/bb/cache_extra.py

@@ -1,54 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
-# Extra RecipeInfo will be all defined in this file. Currently,
-# Only Hob (Image Creator) Requests some extra fields. So
-# HobRecipeInfo is defined. It's named HobRecipeInfo because it
-# is introduced by 'hob'. Users could also introduce other
-# RecipeInfo or simply use those already defined RecipeInfo.
-# In the following patch, this newly defined new extra RecipeInfo
-# will be dynamically loaded and used for loading/saving the extra
-# cache fields  
-
-# Copyright (C) 2011, Intel Corporation. All rights reserved.
-
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-from bb.cache import RecipeInfoCommon
-
-class HobRecipeInfo(RecipeInfoCommon):
-    __slots__ = ()
-
-    classname = "HobRecipeInfo"
-    # please override this member with the correct data cache file
-    # such as (bb_cache.dat, bb_extracache_hob.dat) 
-    cachefile = "bb_extracache_" + classname +".dat"        
-
-    def __init__(self, filename, metadata):
-
-        self.summary = self.getvar('SUMMARY', metadata)
-        self.license = self.getvar('LICENSE', metadata)
-        self.section = self.getvar('SECTION', metadata)
-
-    @classmethod
-    def init_cacheData(cls, cachedata):
-        # CacheData in Hob RecipeInfo Class
-        cachedata.summary = {}
-        cachedata.license = {}
-        cachedata.section = {}
-
-    def add_cacheData(self, cachedata, fn):
-        cachedata.summary[fn] = self.summary
-        cachedata.license[fn] = self.license
-        cachedata.section[fn] = self.section

bitbake/lib/bb/codeparser.py

@@ -1,34 +1,29 @@
+from pysh import pyshyacc, pyshlex
+from itertools import chain
+from bb import msg, utils
 import ast
 import codegen
-import logging
-import os.path
-import bb.utils, bb.data
-from itertools import chain
-from pysh import pyshyacc, pyshlex, sherrors
 
-
-logger = logging.getLogger('BitBake.CodeParser')
 PARSERCACHE_VERSION = 2
 
 try:
     import cPickle as pickle
 except ImportError:
     import pickle
-    logger.info('Importing cPickle failed.  Falling back to a very slow implementation.')
-
+    bb.msg.note(1, bb.msg.domain.Cache, "Importing cPickle failed. Falling back to a very slow implementation.")
 
 def check_indent(codestr):
     """If the code is indented, add a top level piece of code to 'remove' the indentation"""
 
     i = 0
-    while codestr[i] in ["\n", "\t", " "]:
+    while codestr[i] in ["\n", "	", " "]:
         i = i + 1
 
     if i == 0:
         return codestr
 
-    if codestr[i-1] == "\t" or codestr[i-1] == " ":
-        return "if 1:\n" + codestr
+    if codestr[i-1] is "	" or codestr[i-1] is " ":
+        return "if 1:\n" + codestr        
 
     return codestr
 
@@ -36,18 +31,15 @@ pythonparsecache = {}
 shellparsecache = {}
 
 def parser_cachefile(d):
-    cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or
-                bb.data.getVar("CACHE", d, True))
+    cachedir = bb.data.getVar("PERSISTENT_DIR", d, True) or bb.data.getVar("CACHE", d, True)
     if cachedir in [None, '']:
         return None
     bb.utils.mkdirhier(cachedir)
     cachefile = os.path.join(cachedir, "bb_codeparser.dat")
-    logger.debug(1, "Using cache in '%s' for codeparser cache", cachefile)
+    bb.msg.debug(1, bb.msg.domain.Cache, "Using cache in '%s' for codeparser cache" % cachefile)
     return cachefile
 
 def parser_cache_init(d):
-    global pythonparsecache
-    global shellparsecache
 
     cachefile = parser_cachefile(d)
     if not cachefile:
@@ -62,167 +54,129 @@ def parser_cache_init(d):
     if version != PARSERCACHE_VERSION:
         return
 
-    pythonparsecache = data[0]
-    shellparsecache = data[1]
+    bb.codeparser.pythonparsecache = data[0]
+    bb.codeparser.shellparsecache = data[1]
 
 def parser_cache_save(d):
+
     cachefile = parser_cachefile(d)
     if not cachefile:
         return
 
-    glf = bb.utils.lockfile(cachefile + ".lock", shared=True)
-
-    i = os.getpid()
-    lf = None
-    while not lf:
-        shellcache = {}
-        pythoncache = {}
-
-        lf = bb.utils.lockfile(cachefile + ".lock." + str(i), retry=False)
-        if not lf or os.path.exists(cachefile + "-" + str(i)):
-            if lf:
-               bb.utils.unlockfile(lf) 
-               lf = None
-            i = i + 1
-            continue
-
-        try:
-            p = pickle.Unpickler(file(cachefile, "rb"))
-            data, version = p.load()
-        except (IOError, EOFError, ValueError):
-            data, version = None, None
-
-        if version != PARSERCACHE_VERSION:
-            shellcache = shellparsecache
-            pythoncache = pythonparsecache
-        else:
-            for h in pythonparsecache:
-                if h not in data[0]:
-                    pythoncache[h] = pythonparsecache[h]
-            for h in shellparsecache:
-                if h not in data[1]:
-                    shellcache[h] = shellparsecache[h]
-
-        p = pickle.Pickler(file(cachefile + "-" + str(i), "wb"), -1)
-        p.dump([[pythoncache, shellcache], PARSERCACHE_VERSION])
-
-    bb.utils.unlockfile(lf)
-    bb.utils.unlockfile(glf)
-
-def parser_cache_savemerge(d):
-    cachefile = parser_cachefile(d)
-    if not cachefile:
-        return
-
-    glf = bb.utils.lockfile(cachefile + ".lock")
-
-    try:
-        p = pickle.Unpickler(file(cachefile, "rb"))
-        data, version = p.load()
-    except (IOError, EOFError):
-        data, version = None, None
-
-    if version != PARSERCACHE_VERSION:
-        data = [{}, {}]
-
-    for f in [y for y in os.listdir(os.path.dirname(cachefile)) if y.startswith(os.path.basename(cachefile) + '-')]:
-        f = os.path.join(os.path.dirname(cachefile), f)
-        try:
-            p = pickle.Unpickler(file(f, "rb"))
-            extradata, version = p.load()
-        except (IOError, EOFError):
-            extradata, version = [{}, {}], None
-        
-        if version != PARSERCACHE_VERSION:
-            continue
-
-        for h in extradata[0]:
-            if h not in data[0]:
-                data[0][h] = extradata[0][h]
-        for h in extradata[1]:
-            if h not in data[1]:
-                data[1][h] = extradata[1][h]
-        os.unlink(f)
-
     p = pickle.Pickler(file(cachefile, "wb"), -1)
-    p.dump([data, PARSERCACHE_VERSION])
-
-    bb.utils.unlockfile(glf)
-
-
-Logger = logging.getLoggerClass()
-class BufferedLogger(Logger):
-    def __init__(self, name, level=0, target=None):
-        Logger.__init__(self, name)
-        self.setLevel(level)
-        self.buffer = []
-        self.target = target
-
-    def handle(self, record):
-        self.buffer.append(record)
-
-    def flush(self):
-        for record in self.buffer:
-            self.target.handle(record)
-        self.buffer = []
+    p.dump([[bb.codeparser.pythonparsecache, bb.codeparser.shellparsecache], PARSERCACHE_VERSION])
 
 class PythonParser():
-    getvars = ("d.getVar", "bb.data.getVar", "data.getVar")
-    execfuncs = ("bb.build.exec_func", "bb.build.exec_task")
-
-    def warn(self, func, arg):
-        """Warn about calls of bitbake APIs which pass a non-literal
-        argument for the variable name, as we're not able to track such
-        a reference.
+    class ValueVisitor():
+        """Visitor to traverse a python abstract syntax tree and obtain
+        the variables referenced via bitbake metadata APIs, and the external
+        functions called.
         """
 
-        try:
-            funcstr = codegen.to_source(func)
-            argstr = codegen.to_source(arg)
-        except TypeError:
-            self.log.debug(2, 'Failed to convert function and argument to source form')
-        else:
-            self.log.debug(1, self.unhandled_message % (funcstr, argstr))
+        getvars = ("d.getVar", "bb.data.getVar", "data.getVar")
+        expands = ("d.expand", "bb.data.expand", "data.expand")
+        execs = ("bb.build.exec_func", "bb.build.exec_task")
 
-    def visit_Call(self, node):
-        name = self.called_node_name(node.func)
-        if name in self.getvars:
-            if isinstance(node.args[0], ast.Str):
-                self.var_references.add(node.args[0].s)
-            else:
-                self.warn(node.func, node.args[0])
-        elif name in self.execfuncs:
-            if isinstance(node.args[0], ast.Str):
-                self.var_execs.add(node.args[0].s)
-            else:
-                self.warn(node.func, node.args[0])
-        elif name and isinstance(node.func, (ast.Name, ast.Attribute)):
-            self.execs.add(name)
+        @classmethod
+        def _compare_name(cls, strparts, node):
+            """Given a sequence of strings representing a python name,
+            where the last component is the actual Name and the prior
+            elements are Attribute nodes, determine if the supplied node
+            matches.
+            """
 
-    def called_node_name(self, node):
-        """Given a called node, return its original string form"""
-        components = []
-        while node:
+            if not strparts:
+                return True
+
+            current, rest = strparts[0], strparts[1:]
             if isinstance(node, ast.Attribute):
-                components.append(node.attr)
-                node = node.value
+                if current == node.attr:
+                    return cls._compare_name(rest, node.value)
             elif isinstance(node, ast.Name):
-                components.append(node.id)
-                return '.'.join(reversed(components))
+                if current == node.id:
+                    return True
+            return False
+
+        @classmethod
+        def compare_name(cls, value, node):
+            """Convenience function for the _compare_node method, which
+            can accept a string (which is split by '.' for you), or an
+            iterable of strings, in which case it checks to see if any of
+            them match, similar to isinstance.
+            """
+
+            if isinstance(value, basestring):
+                return cls._compare_name(tuple(reversed(value.split("."))),
+                                         node)
             else:
-                break
+                return any(cls.compare_name(item, node) for item in value)
 
-    def __init__(self, name, log):
-        self.var_references = set()
-        self.var_execs = set()
+        def __init__(self, value):
+            self.var_references = set()
+            self.var_execs = set()
+            self.direct_func_calls = set()
+            self.var_expands = set()
+            self.value = value
+
+        @classmethod
+        def warn(cls, func, arg):
+            """Warn about calls of bitbake APIs which pass a non-literal
+            argument for the variable name, as we're not able to track such
+            a reference.
+            """
+
+            try:
+                funcstr = codegen.to_source(func)
+                argstr = codegen.to_source(arg)
+            except TypeError:
+                msg.debug(2, None, "Failed to convert function and argument to source form")
+            else:
+                msg.debug(1, None, "Warning: in call to '%s', argument '%s' is not a literal" %
+                                     (funcstr, argstr))
+
+        def visit_Call(self, node):
+            if self.compare_name(self.getvars, node.func):
+                if isinstance(node.args[0], ast.Str):
+                    self.var_references.add(node.args[0].s)
+                else:
+                    self.warn(node.func, node.args[0])
+            elif self.compare_name(self.expands, node.func):
+                if isinstance(node.args[0], ast.Str):
+                    self.warn(node.func, node.args[0])
+                    self.var_expands.update(node.args[0].s)
+                elif isinstance(node.args[0], ast.Call) and \
+                     self.compare_name(self.getvars, node.args[0].func):
+                    pass
+                else:
+                    self.warn(node.func, node.args[0])
+            elif self.compare_name(self.execs, node.func):
+                if isinstance(node.args[0], ast.Str):
+                    self.var_execs.add(node.args[0].s)
+                else:
+                    self.warn(node.func, node.args[0])
+            elif isinstance(node.func, ast.Name):
+                self.direct_func_calls.add(node.func.id)
+            elif isinstance(node.func, ast.Attribute):
+                # We must have a qualified name.  Therefore we need
+                # to walk the chain of 'Attribute' nodes to determine
+                # the qualification.
+                attr_node = node.func.value
+                identifier = node.func.attr
+                while isinstance(attr_node, ast.Attribute):
+                    identifier = attr_node.attr + "." + identifier
+                    attr_node = attr_node.value
+                if isinstance(attr_node, ast.Name):
+                    identifier = attr_node.id + "." + identifier
+                self.direct_func_calls.add(identifier)
+
+    def __init__(self):
+        #self.funcdefs = set()
         self.execs = set()
+        #self.external_cmds = set()
         self.references = set()
-        self.log = BufferedLogger('BitBake.Data.%s' % name, logging.DEBUG, log)
-
-        self.unhandled_message = "in call of %s, argument '%s' is not a string literal"
-        self.unhandled_message = "while parsing %s, %s" % (name, self.unhandled_message)
 
     def parse_python(self, node):
+
         h = hash(str(node))
 
         if h in pythonparsecache:
@@ -230,28 +184,27 @@ class PythonParser():
             self.execs = pythonparsecache[h]["execs"]
             return
 
-        code = compile(check_indent(str(node)), "<string>", "exec",
+        code = compile(check_indent(str(node)), "<string>", "exec", 
                        ast.PyCF_ONLY_AST)
 
+        visitor = self.ValueVisitor(code)
         for n in ast.walk(code):
             if n.__class__.__name__ == "Call":
-                self.visit_Call(n)
+                visitor.visit_Call(n)
 
-        self.references.update(self.var_references)
-        self.references.update(self.var_execs)
+        self.references.update(visitor.var_references)
+        self.references.update(visitor.var_execs)
+        self.execs = visitor.direct_func_calls
 
         pythonparsecache[h] = {}
         pythonparsecache[h]["refs"] = self.references
         pythonparsecache[h]["execs"] = self.execs
 
 class ShellParser():
-    def __init__(self, name, log):
+    def __init__(self):
         self.funcdefs = set()
         self.allexecs = set()
         self.execs = set()
-        self.log = BufferedLogger('BitBake.Data.%s' % name, logging.DEBUG, log)
-        self.unhandled_template = "unable to handle non-literal command '%s'"
-        self.unhandled_template = "while parsing %s, %s" % (name, self.unhandled_template)
 
     def parse_shell(self, value):
         """Parse the supplied shell code in a string, returning the external
@@ -267,7 +220,7 @@ class ShellParser():
         try:
             tokens, _ = pyshyacc.parse(value, eof=True, debug=False)
         except pyshlex.NeedMore:
-            raise sherrors.ShellSyntaxError("Unexpected EOF")
+            raise ShellSyntaxError("Unexpected EOF")
 
         for token in tokens:
             self.process_tokens(token)
@@ -366,10 +319,11 @@ class ShellParser():
 
                 cmd = word[1]
                 if cmd.startswith("$"):
-                    self.log.debug(1, self.unhandled_template % cmd)
+                    msg.debug(1, None, "Warning: execution of non-literal command '%s'" % cmd)
                 elif cmd == "eval":
                     command = " ".join(word for _, word in words[1:])
                     self.parse_shell(command)
                 else:
                     self.allexecs.add(cmd)
                 break
+

bitbake/lib/bb/command.py

@@ -35,25 +35,12 @@ import bb.data
 async_cmds = {}
 sync_cmds = {}
 
-
-class CommandCompleted(bb.event.Event):
-    pass
-
-class CommandExit(bb.event.Event):
-    def  __init__(self, exitcode):
-        bb.event.Event.__init__(self)
-        self.exitcode = int(exitcode)
-
-class CommandFailed(CommandExit):
-    def __init__(self, message):
-        self.error = message
-        CommandExit.__init__(self, 1)
-
 class Command:
     """
     A queue of asynchronous commands for bitbake
     """
     def __init__(self, cooker):
+
         self.cooker = cooker
         self.cmds_sync = CommandsSync()
         self.cmds_async = CommandsAsync()
@@ -82,7 +69,7 @@ class Command:
             if command not in CommandsAsync.__dict__:
                 return "No such command"
             self.currentAsyncCommand = (command, commandline)
-            self.cooker.server_registration_cb(self.cooker.runCommands, self.cooker)
+            self.cooker.server.register_idle_function(self.cooker.runCommands, self.cooker)
             return True
         except:
             import traceback
@@ -94,8 +81,7 @@ class Command:
                 (command, options) = self.currentAsyncCommand
                 commandmethod = getattr(CommandsAsync, command)
                 needcache = getattr( commandmethod, "needcache" )
-                if (needcache and self.cooker.state in
-                    (bb.cooker.state.initial, bb.cooker.state.parsing)):
+                if needcache and self.cooker.cookerState != bb.cooker.cookerParsed:
                     self.cooker.updateCache()
                     return True
                 else:
@@ -118,13 +104,11 @@ class Command:
             self.finishAsyncCommand(traceback.format_exc())
             return False
 
-    def finishAsyncCommand(self, msg=None, code=None):
-        if msg:
-            bb.event.fire(CommandFailed(msg), self.cooker.configuration.event_data)
-        elif code:
-            bb.event.fire(CommandExit(code), self.cooker.configuration.event_data)
+    def finishAsyncCommand(self, error = None):
+        if error:
+            bb.event.fire(CookerCommandFailed(error), self.cooker.configuration.event_data)
         else:
-            bb.event.fire(CommandCompleted(), self.cooker.configuration.event_data)
+            bb.event.fire(CookerCommandCompleted(), self.cooker.configuration.event_data)
         self.currentAsyncCommand = None
 
 
@@ -139,13 +123,13 @@ class CommandsSync:
         """
         Trigger cooker 'shutdown' mode
         """
-        command.cooker.shutdown()
+        command.cooker.cookerAction = bb.cooker.cookerShutdown
 
     def stateStop(self, command, params):
         """
         Stop the cooker
         """
-        command.cooker.stop()
+        command.cooker.cookerAction = bb.cooker.cookerStop
 
     def getCmdLineAction(self, command, params):
         """
@@ -172,13 +156,6 @@ class CommandsSync:
         value = params[1]
         bb.data.setVar(varname, value, command.cooker.configuration.data)
 
-    def resetCooker(self, command, params):
-        """
-        Reset the cooker to its initial state, thus forcing a reparse for
-        any async command that has the needcache property set to True
-        """
-        command.cooker.reset()
-
 
 class CommandsAsync:
     """
@@ -229,57 +206,6 @@ class CommandsAsync:
         command.finishAsyncCommand()
     generateDotGraph.needcache = True
 
-    def generateTargetsTree(self, command, params):
-        """
-        Generate a tree of buildable targets.
-        If klass is provided ensure all recipes that inherit the class are
-        included in the package list.
-        If pkg_list provided use that list (plus any extras brought in by
-        klass) rather than generating a tree for all packages.
-        """
-        klass = params[0]
-        if len(params) > 1:
-            pkg_list = params[1]
-        else:
-            pkg_list = []
-
-        command.cooker.generateTargetsTree(klass, pkg_list)
-        command.finishAsyncCommand()
-    generateTargetsTree.needcache = True
-
-    def findConfigFiles(self, command, params):
-        """
-        Find config files which provide appropriate values
-        for the passed configuration variable. i.e. MACHINE
-        """
-        varname = params[0]
-
-        command.cooker.findConfigFiles(varname)
-        command.finishAsyncCommand()
-    findConfigFiles.needcache = True
-
-    def findFilesMatchingInDir(self, command, params):
-        """
-        Find implementation files matching the specified pattern
-        in the requested subdirectory of a BBPATH
-        """
-        pattern = params[0]
-        directory = params[1]
-
-        command.cooker.findFilesMatchingInDir(pattern, directory)
-        command.finishAsyncCommand()
-    findFilesMatchingInDir.needcache = True
-
-    def findConfigFilePath(self, command, params):
-        """
-        Find the path of the requested configuration file
-        """
-        configfile = params[0]
-
-        command.cooker.findConfigFilePath(configfile)
-        command.finishAsyncCommand()
-    findConfigFilePath.needcache = False
-
     def showVersions(self, command, params):
         """
         Show the currently selected versions
@@ -318,20 +244,37 @@ class CommandsAsync:
         command.finishAsyncCommand()
     parseFiles.needcache = True
 
-    def reparseFiles(self, command, params):
-        """
-        Reparse .bb files
-        """
-        command.cooker.reparseFiles()
-        command.finishAsyncCommand()
-    reparseFiles.needcache = True
-
     def compareRevisions(self, command, params):
         """
         Parse the .bb files
         """
-        if bb.fetch.fetcher_compare_revisions(command.cooker.configuration.data):
-            command.finishAsyncCommand(code=1)
-        else:
-            command.finishAsyncCommand()
+        command.cooker.compareRevisions()
+        command.finishAsyncCommand()
     compareRevisions.needcache = True
+
+#
+# Events
+#
+class CookerCommandCompleted(bb.event.Event):
+    """
+    Cooker command completed
+    """
+    def  __init__(self):
+        bb.event.Event.__init__(self)
+
+
+class CookerCommandFailed(bb.event.Event):
+    """
+    Cooker command completed
+    """
+    def  __init__(self, error):
+        bb.event.Event.__init__(self)
+        self.error = error
+
+class CookerCommandSetExitCode(bb.event.Event):
+    """
+    Set the exit code for a cooker command
+    """
+    def  __init__(self, exitcode):
+        bb.event.Event.__init__(self)
+        self.exitcode = int(exitcode)

bitbake/lib/bb/compat.py

@@ -1,28 +0,0 @@
-"""Code pulled from future python versions, here for compatibility"""
-
-def total_ordering(cls):
-    """Class decorator that fills in missing ordering methods"""
-    convert = {
-        '__lt__': [('__gt__', lambda self, other: other < self),
-                   ('__le__', lambda self, other: not other < self),
-                   ('__ge__', lambda self, other: not self < other)],
-        '__le__': [('__ge__', lambda self, other: other <= self),
-                   ('__lt__', lambda self, other: not other <= self),
-                   ('__gt__', lambda self, other: not self <= other)],
-        '__gt__': [('__lt__', lambda self, other: other > self),
-                   ('__ge__', lambda self, other: not other > self),
-                   ('__le__', lambda self, other: not self > other)],
-        '__ge__': [('__le__', lambda self, other: other >= self),
-                   ('__gt__', lambda self, other: not other >= self),
-                   ('__lt__', lambda self, other: not self >= other)]
-    }
-    roots = set(dir(cls)) & set(convert)
-    if not roots:
-        raise ValueError('must define at least one ordering operation: < > <= >=')
-    root = max(roots)       # prefer __lt__ to __le__ to __gt__ to __ge__
-    for opname, opfunc in convert[root]:
-        if opname not in roots:
-            opfunc.__name__ = opname
-            opfunc.__doc__ = getattr(int, opname).__doc__
-            setattr(cls, opname, opfunc)
-    return cls

bitbake/lib/bb/data.py

@@ -49,7 +49,6 @@ from bb import data_smart
 from bb import codeparser
 import bb
 
-logger = data_smart.logger
 _dict_type = data_smart.DataSmart
 
 def init():
@@ -160,17 +159,14 @@ def expandKeys(alterdata, readdata = None):
         ekey = todolist[key]
         renameVar(key, ekey, alterdata)
 
-def inheritFromOS(d, savedenv, permitted):
-    """Inherit variables from the initial environment."""
-    exportlist = bb.utils.preserved_envvars_exported()
-    for s in savedenv.keys():
-        if s in permitted:
-            try:
-                setVar(s, getVar(s, savedenv, True), d)
-                if s in exportlist:
-                    setVarFlag(s, "export", True, d)
-            except TypeError:
-                pass
+def inheritFromOS(d):
+    """Inherit variables from the environment."""
+    for s in os.environ.keys():
+        try:
+            setVar(s, os.environ[s], d)
+            setVarFlag(s, "export", True, d)
+        except TypeError:
+            pass
 
 def emit_var(var, o=sys.__stdout__, d = init(), all=False):
     """Emit a variable to be sourced by a shell."""
@@ -189,13 +185,12 @@ def emit_var(var, o=sys.__stdout__, d = init(), all=False):
         val = getVar(var, d, 1)
     except (KeyboardInterrupt, bb.build.FuncFailed):
         raise
-    except Exception as exc:
+    except Exception, exc:
         o.write('# expansion of %s threw %s: %s\n' % (var, exc.__class__.__name__, str(exc)))
         return 0
 
     if all:
-        commentVal = re.sub('\n', '\n#', str(oval))
-        o.write('# %s=%s\n' % (var, commentVal))
+        o.write('# %s=%s\n' % (var, oval))
 
     if (var.find("-") != -1 or var.find(".") != -1 or var.find('{') != -1 or var.find('}') != -1 or var.find('+') != -1) and not all:
         return 0
@@ -204,7 +199,7 @@ def emit_var(var, o=sys.__stdout__, d = init(), all=False):
 
     if unexport:
         o.write('unset %s\n' % varExpanded)
-        return 0
+        return 1
 
     if not val:
         return 0
@@ -222,9 +217,8 @@ def emit_var(var, o=sys.__stdout__, d = init(), all=False):
     # if we're going to output this within doublequotes,
     # to a shell, we need to escape the quotes in the var
     alter = re.sub('"', '\\"', val.strip())
-    alter = re.sub('\n', ' \\\n', alter)
     o.write('%s="%s"\n' % (varExpanded, alter))
-    return 0
+    return 1
 
 def emit_env(o=sys.__stdout__, d = init(), all=False):
     """Emits all items in the data store in a format such that it can be sourced by a shell."""
@@ -236,20 +230,19 @@ def emit_env(o=sys.__stdout__, d = init(), all=False):
         for key in keys:
             emit_var(key, o, d, all and not isfunc) and o.write('\n')
 
-def exported_keys(d):
-    return (key for key in d.keys() if not key.startswith('__') and
-                                      d.getVarFlag(key, 'export') and
-                                      not d.getVarFlag(key, 'unexport'))
-
-def exported_vars(d):
-    for key in exported_keys(d):
+def export_vars(d):
+    keys = (key for key in d.keys() if d.getVarFlag(key, "export"))
+    ret = {}
+    for k in keys:
         try:
-            value = d.getVar(key, True)
-        except Exception:
+            v = d.getVar(k, True)
+            if v: 
+                ret[k] = v
+        except (KeyboardInterrupt, bb.build.FuncFailed):
+            raise
+        except Exception, exc:
             pass
-
-        if value is not None:
-            yield key, str(value)
+    return ret
 
 def emit_func(func, o=sys.__stdout__, d = init()):
     """Emits all items in the data store in a format such that it can be sourced by a shell."""
@@ -258,8 +251,8 @@ def emit_func(func, o=sys.__stdout__, d = init()):
     for key in keys:
         emit_var(key, o, d, False) and o.write('\n')
 
-    emit_var(func, o, d, False) and o.write('\n')
-    newdeps = bb.codeparser.ShellParser(func, logger).parse_shell(d.getVar(func, True))
+    emit_var(func, o, d, False) and o.write('\n')    
+    newdeps = bb.codeparser.ShellParser().parse_shell(d.getVar(func, True))
     seen = set()
     while newdeps:
         deps = newdeps
@@ -268,45 +261,38 @@ def emit_func(func, o=sys.__stdout__, d = init()):
         for dep in deps:
             if bb.data.getVarFlag(dep, "func", d):
                emit_var(dep, o, d, False) and o.write('\n')
-               newdeps |=  bb.codeparser.ShellParser(dep, logger).parse_shell(d.getVar(dep, True))
+               newdeps |=  bb.codeparser.ShellParser().parse_shell(d.getVar(dep, True))
         newdeps -= seen
 
 def update_data(d):
     """Performs final steps upon the datastore, including application of overrides"""
     d.finalize()
 
-def build_dependencies(key, keys, shelldeps, vardepvals, d):
+def build_dependencies(key, keys, shelldeps, d):
     deps = set()
-    vardeps = d.getVarFlag(key, "vardeps", True)
     try:
-        value = d.getVar(key, False)
-        if key in vardepvals:
-           value =  d.getVarFlag(key, "vardepvalue", True)
-        elif d.getVarFlag(key, "func"):
+        if d.getVarFlag(key, "func"):
             if d.getVarFlag(key, "python"):
-                parsedvar = d.expandWithRefs(value, key)
-                parser = bb.codeparser.PythonParser(key, logger)
+                parsedvar = d.expandWithRefs(d.getVar(key, False), key)
+                parser = bb.codeparser.PythonParser()
                 parser.parse_python(parsedvar.value)
                 deps = deps | parser.references
             else:
-                parsedvar = d.expandWithRefs(value, key)
-                parser = bb.codeparser.ShellParser(key, logger)
+                parsedvar = d.expandWithRefs(d.getVar(key, False), key)
+                parser = bb.codeparser.ShellParser()
                 parser.parse_shell(parsedvar.value)
                 deps = deps | shelldeps
-            if vardeps is None:
-                parser.log.flush()
             deps = deps | parsedvar.references
             deps = deps | (keys & parser.execs) | (keys & parsedvar.execs)
         else:
-            parser = d.expandWithRefs(value, key)
+            parser = d.expandWithRefs(d.getVar(key, False), key)
             deps |= parser.references
             deps = deps | (keys & parser.execs)
-        deps |= set((vardeps or "").split())
-        deps -= set((d.getVarFlag(key, "vardepsexclude", True) or "").split())
+        deps |= set((d.getVarFlag(key, "vardeps") or "").split())
     except:
-        bb.note("Error expanding variable %s" % key)
+        bb.note("Error expanding variable %s" % key) 
         raise
-    return deps, value
+    return deps
     #bb.note("Variable %s references %s and calls %s" % (key, str(deps), str(execs)))
     #d.setVarFlag(key, "vardeps", deps)
 
@@ -314,14 +300,14 @@ def generate_dependencies(d):
 
     keys = set(key for key in d.keys() if not key.startswith("__"))
     shelldeps = set(key for key in keys if d.getVarFlag(key, "export") and not d.getVarFlag(key, "unexport"))
-    vardepvals = set(key for key in keys if d.getVarFlag(key, "vardepvalue"))
 
     deps = {}
-    values = {}
+    taskdeps = {}
 
     tasklist = bb.data.getVar('__BBTASKS', d) or []
     for task in tasklist:
-        deps[task], values[task] = build_dependencies(task, keys, shelldeps, vardepvals, d)
+        deps[task] = build_dependencies(task, keys, shelldeps, d)
+
         newdeps = deps[task]
         seen = set()
         while newdeps:
@@ -330,11 +316,12 @@ def generate_dependencies(d):
             newdeps = set()
             for dep in nextdeps:
                 if dep not in deps:
-                    deps[dep], values[dep] = build_dependencies(dep, keys, shelldeps, vardepvals, d)
+                    deps[dep] = build_dependencies(dep, keys, shelldeps, d)
                 newdeps |=  deps[dep]
             newdeps -= seen
+        taskdeps[task] = seen | newdeps
         #print "For %s: %s" % (task, str(taskdeps[task]))
-    return tasklist, deps, values
+    return taskdeps, deps
 
 def inherits_class(klass, d):
     val = getVar('__inherit_cache', d) or []

bitbake/lib/bb/data_smart.py

@@ -28,21 +28,17 @@ BitBake build tools.
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 # Based on functions from the base bb module, Copyright 2003 Holger Schurig
 
-import copy, re
-from collections import MutableMapping
-import logging
-import bb, bb.codeparser
+import copy, re, sys
+import bb
 from bb   import utils
 from bb.COW  import COWDictBase
 
-logger = logging.getLogger("BitBake.Data")
 
 __setvar_keyword__ = ["_append", "_prepend"]
 __setvar_regexp__ = re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend)(_(?P<add>.*))?')
 __expand_var_regexp__ = re.compile(r"\${[^{}]+}")
 __expand_python_regexp__ = re.compile(r"\${@.+?}")
 
-
 class VariableParse:
     def __init__(self, varname, d, val = None):
         self.varname = varname
@@ -68,46 +64,16 @@ class VariableParse:
             code = match.group()[3:-1]
             codeobj = compile(code.strip(), self.varname or "<expansion>", "eval")
 
-            parser = bb.codeparser.PythonParser(self.varname, logger)
+            parser = bb.codeparser.PythonParser()
             parser.parse_python(code)
-            if self.varname:
-                vardeps = self.d.getVarFlag(self.varname, "vardeps", True)
-                if vardeps is None:
-                    parser.log.flush()
-            else:
-                parser.log.flush()
             self.references |= parser.references
             self.execs |= parser.execs
 
-            value = utils.better_eval(codeobj, DataContext(self.d))
+            value = utils.better_eval(codeobj, {"d": self.d})
             return str(value)
 
 
-class DataContext(dict):
-    def __init__(self, metadata, **kwargs):
-        self.metadata = metadata
-        dict.__init__(self, **kwargs)
-        self['d'] = metadata
-
-    def __missing__(self, key):
-        value = self.metadata.getVar(key, True)
-        if value is None or self.metadata.getVarFlag(key, 'func'):
-            raise KeyError(key)
-        else:
-            return value
-
-class ExpansionError(Exception):
-    def __init__(self, varname, expression, exception):
-        self.expression = expression
-        self.variablename = varname
-        self.exception = exception
-        self.msg = "Failure expanding variable %s, expression was %s which triggered exception %s: %s" % (varname, expression, type(exception).__name__, exception)
-        Exception.__init__(self, self.msg)
-        self.args = (varname, expression, exception)
-    def __str__(self):
-        return self.msg
-
-class DataSmart(MutableMapping):
+class DataSmart:
     def __init__(self, special = COWDictBase.copy(), seen = COWDictBase.copy() ):
         self.dict = {}
 
@@ -134,10 +100,11 @@ class DataSmart(MutableMapping):
                 s = __expand_python_regexp__.sub(varparse.python_sub, s)
                 if s == olds:
                     break
-            except ExpansionError:
+            except KeyboardInterrupt:
+                raise
+            except:
+                bb.msg.note(1, bb.msg.domain.Data, "%s:%s while evaluating:\n%s" % (sys.exc_info()[0], sys.exc_info()[1], s))
                 raise
-            except Exception as exc:
-                raise ExpansionError(varname, s, exc)
 
         varparse.value = s
 
@@ -148,7 +115,7 @@ class DataSmart(MutableMapping):
 
     def expand(self, s, varname):
         return self.expandWithRefs(s, varname).value
-
+    
 
     def finalize(self):
         """Performs final steps upon the datastore, including application of overrides"""
@@ -178,39 +145,43 @@ class DataSmart(MutableMapping):
             if o not in self._seen_overrides:
                 continue
 
-            vars = self._seen_overrides[o].copy()
+            vars = self._seen_overrides[o]
             for var in vars:
                 name = var[:-l]
                 try:
-                    self.setVar(name, self.getVar(var, False))
-                    self.delVar(var)
+                    self[name] = self[var]
                 except Exception:
-                    logger.info("Untracked delVar")
+                    bb.msg.note(1, bb.msg.domain.Data, "Untracked delVar")
 
         # now on to the appends and prepends
-        for op in __setvar_keyword__:
-            if op in self._special_values:
-                appends = self._special_values[op] or []
-                for append in appends:
-                    keep = []
-                    for (a, o) in self.getVarFlag(append, op) or []:
-                        if o and not o in overrides:
-                            keep.append((a ,o))
-                            continue
+        if "_append" in self._special_values:
+            appends = self._special_values["_append"] or []
+            for append in appends:
+                for (a, o) in self.getVarFlag(append, "_append") or []:
+                    # maybe the OVERRIDE was not yet added so keep the append
+                    if (o and o in overrides) or not o:
+                        self.delVarFlag(append, "_append")
+                    if o and not o in overrides:
+                        continue
 
-                        if op == "_append":
-                            sval = self.getVar(append, False) or ""
-                            sval += a
-                            self.setVar(append, sval)
-                        elif op == "_prepend":
-                            sval = a + (self.getVar(append, False) or "")
-                            self.setVar(append, sval)
+                    sval = self.getVar(append, False) or ""
+                    sval += a
+                    self.setVar(append, sval)
 
-                    # We save overrides that may be applied at some later stage
-                    if keep:
-                        self.setVarFlag(append, op, keep)
-                    else:
-                        self.delVarFlag(append, op)
+
+        if "_prepend" in self._special_values:
+            prepends = self._special_values["_prepend"] or []
+
+            for prepend in prepends:
+                for (a, o) in self.getVarFlag(prepend, "_prepend") or []:
+                    # maybe the OVERRIDE was not yet added so keep the prepend
+                    if (o and o in overrides) or not o:
+                        self.delVarFlag(prepend, "_prepend")
+                    if o and not o in overrides:
+                        continue
+
+                    sval = a + (self.getVar(prepend, False) or "")
+                    self.setVar(prepend, sval)
 
     def initVar(self, var):
         self.expand_cache = {}
@@ -265,19 +236,17 @@ class DataSmart(MutableMapping):
         # more cookies for the cookie monster
         if '_' in var:
             override = var[var.rfind('_')+1:]
-            if len(override) > 0:
-                if override not in self._seen_overrides:
-                    self._seen_overrides[override] = set()
-                self._seen_overrides[override].add( var )
+            if override not in self._seen_overrides:
+                self._seen_overrides[override] = set()
+            self._seen_overrides[override].add( var )
 
         # setting var
         self.dict[var]["content"] = value
 
-    def getVar(self, var, expand=False, noweakdefault=False):
-        value = self.getVarFlag(var, "content", False, noweakdefault)
+    def getVar(self, var, exp):
+        value = self.getVarFlag(var, "content")
 
-        # Call expand() separately to make use of the expand cache
-        if expand and value:
+        if exp and value:
             return self.expand(value, var)
         return value
 
@@ -307,27 +276,18 @@ class DataSmart(MutableMapping):
     def delVar(self, var):
         self.expand_cache = {}
         self.dict[var] = {}
-        if '_' in var:
-            override = var[var.rfind('_')+1:]
-            if override and override in self._seen_overrides and var in self._seen_overrides[override]:
-                self._seen_overrides[override].remove(var)
 
     def setVarFlag(self, var, flag, flagvalue):
         if not var in self.dict:
             self._makeShadowCopy(var)
         self.dict[var][flag] = flagvalue
 
-    def getVarFlag(self, var, flag, expand=False, noweakdefault=False):
+    def getVarFlag(self, var, flag):
         local_var = self._findVar(var)
-        value = None
         if local_var:
             if flag in local_var:
-                value = copy.copy(local_var[flag])
-            elif flag == "content" and "defaultval" in local_var and not noweakdefault:
-                value = copy.copy(local_var["defaultval"])
-        if expand and value:
-            value = self.expand(value, None)
-        return value
+                return copy.copy(local_var[flag])
+        return None
 
     def delVarFlag(self, var, flag):
         local_var = self._findVar(var)
@@ -389,57 +349,23 @@ class DataSmart(MutableMapping):
 
         return data
 
-    def expandVarref(self, variable, parents=False):
-        """Find all references to variable in the data and expand it
-           in place, optionally descending to parent datastores."""
-
-        if parents:
-            keys = iter(self)
-        else:
-            keys = self.localkeys()
-
-        ref = '${%s}' % variable
-        value = self.getVar(variable, False)
-        for key in keys:
-            referrervalue = self.getVar(key, False)
-            if referrervalue and ref in referrervalue:
-                self.setVar(key, referrervalue.replace(ref, value))
-
-    def localkeys(self):
-        for key in self.dict:
-            if key != '_data':
-                yield key
-
-    def __iter__(self):
-        def keylist(d):        
-            klist = set()
-            for key in d:
-                if key == "_data":
-                    continue
-                if not d[key]:
-                    continue
-                klist.add(key)
-
+    # Dictionary Methods
+    def keys(self):
+        def _keys(d, mykey):
             if "_data" in d:
-                klist |= keylist(d["_data"])
+                _keys(d["_data"], mykey)
 
-            return klist
-
-        for k in keylist(self.dict):
-             yield k
-
-    def __len__(self):
-        return len(frozenset(self))
+            for key in d.keys():
+                if key != "_data":
+                    mykey[key] = None
+        keytab = {}
+        _keys(self.dict, keytab)
+        return keytab.keys()
 
     def __getitem__(self, item):
-        value = self.getVar(item, False)
-        if value is None:
-            raise KeyError(item)
-        else:
-            return value
+        #print "Warning deprecated"
+        return self.getVar(item, False)
 
-    def __setitem__(self, var, value):
-        self.setVar(var, value)
-
-    def __delitem__(self, var):
-        self.delVar(var)
+    def __setitem__(self, var, data):
+        #print "Warning deprecated"
+        self.setVar(var, data)

bitbake/lib/bb/event.py

@@ -24,23 +24,16 @@ BitBake build tools.
 
 import os, sys
 import warnings
-try:
-    import cPickle as pickle
-except ImportError:
-    import pickle
-import logging
-import atexit
-import traceback
 import bb.utils
+import pickle
 
 # This is the pid for which we should generate the event. This is set when
 # the runqueue forks off.
 worker_pid = 0
 worker_pipe = None
+useStdout = True
 
-logger = logging.getLogger('BitBake.Event')
-
-class Event(object):
+class Event:
     """Base class for events"""
 
     def __init__(self):
@@ -61,59 +54,26 @@ _ui_handler_seq = 0
 bb.utils._context["NotHandled"] = NotHandled
 bb.utils._context["Handled"] = Handled
 
-def execute_handler(name, handler, event, d):
-    event.data = d
-    try:
-        ret = handler(event)
-    except bb.parse.SkipPackage:
-        raise
-    except Exception:
-        etype, value, tb = sys.exc_info()
-        logger.error("Execution of event handler '%s' failed" % name,
-                        exc_info=(etype, value, tb.tb_next))
-        raise
-    except SystemExit as exc:
-        if exc.code != 0:
-            logger.error("Execution of event handler '%s' failed" % name)
-        raise
-    finally:
+def fire_class_handlers(event, d):
+    import bb.msg
+    if isinstance(event, bb.msg.MsgBase):
+        return
+
+    for handler in _handlers:
+        h = _handlers[handler]
+        event.data = d
+        if type(h).__name__ == "code":
+            locals = {"e": event}
+            bb.utils.simple_exec(h, locals)
+            ret = bb.utils.better_eval("tmpHandler(e)", locals)
+            if ret is not None:
+                warnings.warn("Using Handled/NotHandled in event handlers is deprecated",
+                              DeprecationWarning, stacklevel = 2)
+        else:
+            h(event)
         del event.data
 
-    if ret is not None:
-        warnings.warn("Using Handled/NotHandled in event handlers is deprecated",
-                        DeprecationWarning, stacklevel = 2)
-
-def fire_class_handlers(event, d):
-    if isinstance(event, logging.LogRecord):
-        return
-
-    for name, handler in _handlers.iteritems():
-        try:
-            execute_handler(name, handler, event, d)
-        except Exception:
-            continue
-
-ui_queue = []
-@atexit.register
-def print_ui_queue():
-    """If we're exiting before a UI has been spawned, display any queued
-    LogRecords to the console."""
-    logger = logging.getLogger("BitBake")
-    if not _ui_handlers:
-        from bb.msg import BBLogFormatter
-        console = logging.StreamHandler(sys.stdout)
-        console.setFormatter(BBLogFormatter("%(levelname)s: %(message)s"))
-        logger.handlers = [console]
-        for event in ui_queue:
-            if isinstance(event, logging.LogRecord):
-                logger.handle(event)
-
 def fire_ui_handlers(event, d):
-    if not _ui_handlers:
-        # No UI handlers registered yet, queue up the messages
-        ui_queue.append(event)
-        return
-
     errors = []
     for h in _ui_handlers:
         #print "Sending event %s" % event
@@ -121,10 +81,7 @@ def fire_ui_handlers(event, d):
              # We use pickle here since it better handles object instances
              # which xmlrpc's marshaller does not. Events *must* be serializable
              # by pickle.
-             if hasattr(_ui_handlers[h].event, "sendpickle"):
-                _ui_handlers[h].event.sendpickle((pickle.dumps(event)))
-             else:
-                _ui_handlers[h].event.send(event)
+            _ui_handlers[h].event.send((pickle.dumps(event)))
         except:
             errors.append(h)
     for h in errors:
@@ -147,15 +104,16 @@ def fire(event, d):
 def worker_fire(event, d):
     data = "<event>" + pickle.dumps(event) + "</event>"
     worker_pipe.write(data)
+    worker_pipe.flush()
 
 def fire_from_worker(event, d):
     if not event.startswith("<event>") or not event.endswith("</event>"):
         print("Error, not an event %s" % event)
         return
+    #print "Got event %s" % event
     event = pickle.loads(event[7:-8])
     fire_ui_handlers(event, d)
 
-noop = lambda _: None
 def register(name, handler):
     """Register an Event handler"""
 
@@ -165,19 +123,10 @@ def register(name, handler):
 
     if handler is not None:
         # handle string containing python code
-        if isinstance(handler, basestring):
-            tmp = "def %s(e):\n%s" % (name, handler)
-            try:
-                code = compile(tmp, "%s(e)" % name, "exec")
-            except SyntaxError:
-                logger.error("Unable to register event handler '%s':\n%s", name,
-                             ''.join(traceback.format_exc(limit=0)))
-                _handlers[name] = noop
-                return
-            env = {}
-            bb.utils.simple_exec(code, env)
-            func = bb.utils.better_eval(name, env)
-            _handlers[name] = func
+        if type(handler).__name__ == "str":
+            tmp = "def tmpHandler(e):\n%s" % handler
+            comp = bb.utils.better_compile(tmp, "tmpHandler(e)", "bb.event._registerCode")
+            _handlers[name] = comp
         else:
             _handlers[name] = handler
 
@@ -190,6 +139,7 @@ def remove(name, handler):
 def register_UIHhandler(handler):
     bb.event._ui_handler_seq = bb.event._ui_handler_seq + 1
     _ui_handlers[_ui_handler_seq] = handler
+    bb.event.useStdout = False
     return _ui_handler_seq
 
 def unregister_UIHhandler(handlerNum):
@@ -207,17 +157,13 @@ def getName(e):
 class ConfigParsed(Event):
     """Configuration Parsing Complete"""
 
-class RecipeEvent(Event):
+class RecipeParsed(Event):
+    """ Recipe Parsing Complete """
+
     def __init__(self, fn):
         self.fn = fn
         Event.__init__(self)
 
-class RecipePreFinalise(RecipeEvent):
-    """ Recipe Parsing Complete but not yet finialised"""
-
-class RecipeParsed(RecipeEvent):
-    """ Recipe Parsing Complete """
-
 class StampUpdate(Event):
     """Trigger for any adjustment of the stamp files to happen"""
 
@@ -289,12 +235,11 @@ class BuildCompleted(BuildBase):
 class NoProvider(Event):
     """No Provider for an Event"""
 
-    def __init__(self, item, runtime=False, dependees=None, reasons=[]):
+    def __init__(self, item, runtime=False, dependees=None):
         Event.__init__(self)
         self._item = item
         self._runtime = runtime
         self._dependees = dependees
-        self._reasons = reasons
 
     def getItem(self):
         return self._item
@@ -329,14 +274,10 @@ class MultipleProviders(Event):
         """
         return self._candidates
 
-class ParseStarted(Event):
-    """Recipe parsing for the runqueue has begun"""
-    def __init__(self, total):
-        Event.__init__(self)
-        self.total = total
-
-class ParseCompleted(Event):
-    """Recipe parsing for the runqueue has completed"""
+class ParseProgress(Event):
+    """
+    Parsing Progress Event
+    """
 
     def __init__(self, cached, parsed, skipped, masked, virtuals, errors, total):
         Event.__init__(self)
@@ -349,32 +290,6 @@ class ParseCompleted(Event):
         self.sofar = cached + parsed
         self.total = total
 
-class ParseProgress(Event):
-    """Recipe parsing progress"""
-
-    def __init__(self, current):
-        self.current = current
-
-class CacheLoadStarted(Event):
-    """Loading of the dependency cache has begun"""
-    def __init__(self, total):
-        Event.__init__(self)
-        self.total = total
-
-class CacheLoadProgress(Event):
-    """Cache loading progress"""
-    def __init__(self, current):
-        Event.__init__(self)
-        self.current = current
-
-class CacheLoadCompleted(Event):
-    """Cache loading is complete"""
-    def __init__(self, total, num_entries):
-        Event.__init__(self)
-        self.total = total
-        self.num_entries = num_entries
-
-
 class DepTreeGenerated(Event):
     """
     Event when a dependency tree has been generated
@@ -383,79 +298,3 @@ class DepTreeGenerated(Event):
     def __init__(self, depgraph):
         Event.__init__(self)
         self._depgraph = depgraph
-
-class TargetsTreeGenerated(Event):
-    """
-    Event when a set of buildable targets has been generated
-    """
-    def __init__(self, model):
-        Event.__init__(self)
-        self._model = model
-
-class FilesMatchingFound(Event):
-    """
-    Event when a list of files matching the supplied pattern has
-    been generated
-    """
-    def __init__(self, pattern, matches):
-        Event.__init__(self)
-        self._pattern = pattern
-        self._matches = matches
-
-class ConfigFilesFound(Event):
-    """
-    Event when a list of appropriate config files has been generated
-    """
-    def __init__(self, variable, values):
-        Event.__init__(self)
-        self._variable = variable
-        self._values = values
-
-class ConfigFilePathFound(Event):
-    """
-    Event when a path for a config file has been found
-    """
-    def __init__(self, path):
-        Event.__init__(self)
-        self._path = path
-
-class MsgBase(Event):
-    """Base class for messages"""
-
-    def __init__(self, msg):
-        self._message = msg
-        Event.__init__(self)
-
-class MsgDebug(MsgBase):
-    """Debug Message"""
-
-class MsgNote(MsgBase):
-    """Note Message"""
-
-class MsgWarn(MsgBase):
-    """Warning Message"""
-
-class MsgError(MsgBase):
-    """Error Message"""
-
-class MsgFatal(MsgBase):
-    """Fatal Message"""
-
-class MsgPlain(MsgBase):
-    """General output"""
-
-class LogHandler(logging.Handler):
-    """Dispatch logging messages as bitbake events"""
-
-    def emit(self, record):
-        if record.exc_info:
-            etype, value, tb = record.exc_info
-            if hasattr(tb, 'tb_next'):
-                tb = list(bb.exceptions.extract_traceback(tb, context=3))
-            record.bb_exc_info = (etype, value, tb)
-            record.exc_info = None
-        fire(record, None)
-
-    def filter(self, record):
-        record.taskpid = worker_pid
-        return True

bitbake/lib/bb/exceptions.py

@@ -1,84 +0,0 @@
-from __future__ import absolute_import
-import inspect
-import traceback
-import bb.namedtuple_with_abc
-from collections import namedtuple
-
-
-class TracebackEntry(namedtuple.abc):
-    """Pickleable representation of a traceback entry"""
-    _fields = 'filename lineno function args code_context index'
-    _header = '  File "{0.filename}", line {0.lineno}, in {0.function}{0.args}'
-
-    def format(self, formatter=None):
-        if not self.code_context:
-            return self._header.format(self) + '\n'
-
-        formatted = [self._header.format(self) + ':\n']
-
-        for lineindex, line in enumerate(self.code_context):
-            if formatter:
-                line = formatter(line)
-
-            if lineindex == self.index:
-                formatted.append('    >%s' % line)
-            else:
-                formatted.append('     %s' % line)
-        return formatted
-
-    def __str__(self):
-        return ''.join(self.format())
-
-def _get_frame_args(frame):
-    """Get the formatted arguments and class (if available) for a frame"""
-    arginfo = inspect.getargvalues(frame)
-    if not arginfo.args:
-        return '', None
-
-    firstarg = arginfo.args[0]
-    if firstarg == 'self':
-        self = arginfo.locals['self']
-        cls = self.__class__.__name__
-
-        arginfo.args.pop(0)
-        del arginfo.locals['self']
-    else:
-        cls = None
-
-    formatted = inspect.formatargvalues(*arginfo)
-    return formatted, cls
-
-def extract_traceback(tb, context=1):
-    frames = inspect.getinnerframes(tb, context)
-    for frame, filename, lineno, function, code_context, index in frames:
-        formatted_args, cls = _get_frame_args(frame)
-        if cls:
-            function = '%s.%s' % (cls, function)
-        yield TracebackEntry(filename, lineno, function, formatted_args,
-                             code_context, index)
-
-def format_extracted(extracted, formatter=None, limit=None):
-    if limit:
-        extracted = extracted[-limit:]
-
-    formatted = []
-    for tracebackinfo in extracted:
-        formatted.extend(tracebackinfo.format(formatter))
-    return formatted
-
-
-def format_exception(etype, value, tb, context=1, limit=None, formatter=None):
-    formatted = ['Traceback (most recent call last):\n']
-
-    if hasattr(tb, 'tb_next'):
-        tb = extract_traceback(tb, context)
-
-    formatted.extend(format_extracted(tb, formatter, limit))
-    formatted.extend(traceback.format_exception_only(etype, value))
-    return formatted
-
-def to_string(exc):
-    if isinstance(exc, SystemExit):
-        if not isinstance(exc.code, basestring):
-            return 'Exited with "%d"' % exc.code
-    return str(exc)

bitbake/lib/bb/fetch/__init__.py

@@ -27,15 +27,9 @@ BitBake build tools.
 from __future__ import absolute_import
 from __future__ import print_function
 import os, re
-import logging
 import bb
 from   bb import data
 from   bb import persist_data
-from   bb import utils
-
-__version__ = "1"
-
-logger = logging.getLogger("BitBake.Fetch")
 
 class MalformedUrl(Exception):
     """Exception raised when encountering an invalid url"""
@@ -123,8 +117,9 @@ def encodeurl(decoded):
     return url
 
 def uri_replace(uri, uri_find, uri_replace, d):
+#   bb.msg.note(1, bb.msg.domain.Fetcher, "uri_replace: operating on %s" % uri)
     if not uri or not uri_find or not uri_replace:
-        logger.debug(1, "uri_replace: passed an undefined value, not replacing")
+        bb.msg.debug(1, bb.msg.domain.Fetcher, "uri_replace: passed an undefined value, not replacing")
     uri_decoded = list(decodeurl(uri))
     uri_find_decoded = list(decodeurl(uri_find))
     uri_replace_decoded = list(decodeurl(uri_replace))
@@ -139,32 +134,38 @@ def uri_replace(uri, uri_find, uri_replace, d):
                     if d:
                         localfn = bb.fetch.localpath(uri, d)
                         if localfn:
-                            result_decoded[loc] = os.path.join(os.path.dirname(result_decoded[loc]), os.path.basename(bb.fetch.localpath(uri, d)))
+                            result_decoded[loc] = os.path.dirname(result_decoded[loc]) + "/" + os.path.basename(bb.fetch.localpath(uri, d))
+#                       bb.msg.note(1, bb.msg.domain.Fetcher, "uri_replace: matching %s against %s and replacing with %s" % (i, uri_decoded[loc], uri_replace_decoded[loc]))
             else:
+#               bb.msg.note(1, bb.msg.domain.Fetcher, "uri_replace: no match")
                 return uri
+#           else:
+#               for j in i:
+#                   FIXME: apply replacements against options
     return encodeurl(result_decoded)
 
 methods = []
 urldata_cache = {}
 saved_headrevs = {}
+persistent_database_connection = {}
 
 def fetcher_init(d):
     """
     Called to initialize the fetchers once the configuration data is known.
     Calls before this must not hit the cache.
     """
+    pd = persist_data.PersistData(d, persistent_database_connection)
     # When to drop SCM head revisions controlled by user policy
     srcrev_policy = bb.data.getVar('BB_SRCREV_POLICY', d, 1) or "clear"
     if srcrev_policy == "cache":
-        logger.debug(1, "Keeping SRCREV cache due to cache policy of: %s", srcrev_policy)
+        bb.msg.debug(1, bb.msg.domain.Fetcher, "Keeping SRCREV cache due to cache policy of: %s" % srcrev_policy)
     elif srcrev_policy == "clear":
-        logger.debug(1, "Clearing SRCREV cache due to cache policy of: %s", srcrev_policy)
-        revs = persist_data.persist('BB_URI_HEADREVS', d)
+        bb.msg.debug(1, bb.msg.domain.Fetcher, "Clearing SRCREV cache due to cache policy of: %s" % srcrev_policy)
         try:
-            bb.fetch.saved_headrevs = revs.items()
+            bb.fetch.saved_headrevs = pd.getKeyValues("BB_URI_HEADREVS")
         except:
             pass
-        revs.clear()
+        pd.delDomain("BB_URI_HEADREVS")
     else:
         raise FetchError("Invalid SRCREV cache policy of: %s" % srcrev_policy)
 
@@ -172,23 +173,28 @@ def fetcher_init(d):
         if hasattr(m, "init"):
             m.init(d)
 
-def fetcher_compare_revisions(d):
+    # Make sure our domains exist
+    pd.addDomain("BB_URI_HEADREVS")
+    pd.addDomain("BB_URI_LOCALCOUNT")
+
+def fetcher_compare_revisons(d):
     """
     Compare the revisions in the persistant cache with current values and
     return true/false on whether they've changed.
     """
 
-    data = persist_data.persist('BB_URI_HEADREVS', d).items()
+    pd = persist_data.PersistData(d, persistent_database_connection)
+    data = pd.getKeyValues("BB_URI_HEADREVS")
     data2 = bb.fetch.saved_headrevs
 
     changed = False
     for key in data:
         if key not in data2 or data2[key] != data[key]:
-            logger.debug(1, "%s changed", key)
+            bb.msg.debug(1, bb.msg.domain.Fetcher, "%s changed" % key)
             changed = True
             return True
         else:
-            logger.debug(2, "%s did not change", key)
+            bb.msg.debug(2, bb.msg.domain.Fetcher, "%s did not change" % key)
     return False
 
 # Function call order is usually:
@@ -219,41 +225,11 @@ def init(urls, d, setup = True):
 def mirror_from_string(data):
     return [ i.split() for i in (data or "").replace('\\n','\n').split('\n') if i ]
 
-def verify_checksum(u, ud, d):
-    """
-    verify the MD5 and SHA256 checksum for downloaded src
-
-    return value:
-        - True: checksum matched
-        - False: checksum unmatched
-
-    if checksum is missing in recipes file, "BB_STRICT_CHECKSUM" decide the return value.
-    if BB_STRICT_CHECKSUM = "1" then return false as unmatched, otherwise return true as
-    matched
-    """
-
-    if not ud.type in ["http", "https", "ftp", "ftps"]:
-        return
-
-    md5data = bb.utils.md5_file(ud.localpath)
-    sha256data = bb.utils.sha256_file(ud.localpath)
-
-    if (ud.md5_expected == None or ud.sha256_expected == None):
-        logger.warn('Missing SRC_URI checksum for %s, consider adding to the recipe:\n'
-                    'SRC_URI[%s] = "%s"\nSRC_URI[%s] = "%s"',
-                    ud.localpath, ud.md5_name, md5data,
-                    ud.sha256_name, sha256data)
-        if bb.data.getVar("BB_STRICT_CHECKSUM", d, True) == "1":
-            raise FetchError("No checksum specified for %s." % u)
-        return
-
-    if (ud.md5_expected != md5data or ud.sha256_expected != sha256data):
-        logger.error('The checksums for "%s" did not match.\n'
-                     '  MD5: expected "%s", got "%s"\n'
-                     '  SHA256: expected "%s", got "%s"\n',
-                     ud.localpath, ud.md5_expected, md5data,
-                     ud.sha256_expected, sha256data)
-        raise FetchError("%s checksum mismatch." % u)
+def removefile(f):
+    try:
+        os.remove(f)
+    except:
+        pass
 
 def go(d, urls = None):
     """
@@ -289,7 +265,7 @@ def go(d, urls = None):
                 localpath = ud.localpath
             except FetchError:
                 # Remove any incomplete file
-                bb.utils.remove(ud.localpath)
+                removefile(ud.localpath)
                 # Finally, try fetching uri, u, from MIRRORS
                 mirrors = mirror_from_string(bb.data.getVar('MIRRORS', d, True))
                 localpath = try_mirrors (d, u, mirrors)
@@ -297,7 +273,6 @@ def go(d, urls = None):
                     raise FetchError("Unable to fetch URL %s from any source." % u)
 
         ud.localpath = localpath
-
         if os.path.exists(ud.md5):
             # Touch the md5 file to show active use of the download
             try:
@@ -306,26 +281,21 @@ def go(d, urls = None):
                 # Errors aren't fatal here
                 pass
         else:
-            # Only check the checksums if we've not seen this item before
-            verify_checksum(u, ud, d)
             Fetch.write_md5sum(u, ud, d)
 
         bb.utils.unlockfile(lf)
 
-def checkstatus(d, urls = None):
+def checkstatus(d):
     """
     Check all urls exist upstream
     init must have previously been called
     """
     urldata = init([], d, True)
 
-    if not urls:
-        urls = urldata
-
-    for u in urls:
+    for u in urldata:
         ud = urldata[u]
         m = ud.method
-        logger.debug(1, "Testing URL %s", u)
+        bb.msg.debug(1, bb.msg.domain.Fetcher, "Testing URL %s" % u)
         # First try checking uri, u, from PREMIRRORS
         mirrors = mirror_from_string(bb.data.getVar('PREMIRRORS', d, True))
         ret = try_mirrors(d, u, mirrors, True)
@@ -356,9 +326,6 @@ def localpaths(d):
 
 srcrev_internal_call = False
 
-def get_autorev(d):
-    return get_srcrev(d)
-
 def get_srcrev(d):
     """
     Return the version string for the current package
@@ -382,17 +349,17 @@ def get_srcrev(d):
 
     scms = []
 
-    # Only call setup_localpath on URIs which supports_srcrev()
+    # Only call setup_localpath on URIs which suppports_srcrev()
     urldata = init(bb.data.getVar('SRC_URI', d, 1).split(), d, False)
     for u in urldata:
         ud = urldata[u]
-        if ud.method.supports_srcrev():
+        if ud.method.suppports_srcrev():
             if not ud.setup:
                 ud.setup_localpath(d)
             scms.append(u)
 
     if len(scms) == 0:
-        logger.error("SRCREV was used yet no valid SCM was found in SRC_URI")
+        bb.msg.error(bb.msg.domain.Fetcher, "SRCREV was used yet no valid SCM was found in SRC_URI")
         raise ParameterError
 
     if bb.data.getVar('BB_SRCREV_POLICY', d, True) != "cache":
@@ -406,7 +373,7 @@ def get_srcrev(d):
     #
     format = bb.data.getVar('SRCREV_FORMAT', d, 1)
     if not format:
-        logger.error("The SRCREV_FORMAT variable must be set when multiple SCMs are used.")
+        bb.msg.error(bb.msg.domain.Fetcher, "The SRCREV_FORMAT variable must be set when multiple SCMs are used.")
         raise ParameterError
 
     for scm in scms:
@@ -441,14 +408,14 @@ def runfetchcmd(cmd, d, quiet = False):
     exportvars = ['PATH', 'GIT_PROXY_COMMAND', 'GIT_PROXY_HOST',
                   'GIT_PROXY_PORT', 'GIT_CONFIG', 'http_proxy', 'ftp_proxy',
                   'https_proxy', 'no_proxy', 'ALL_PROXY', 'all_proxy',
-                  'KRB5CCNAME', 'SSH_AUTH_SOCK', 'SSH_AGENT_PID', 'HOME']
+                  'SSH_AUTH_SOCK', 'SSH_AGENT_PID', 'HOME']
 
     for var in exportvars:
         val = data.getVar(var, d, True)
         if val:
             cmd = 'export ' + var + '=\"%s\"; %s' % (val, cmd)
 
-    logger.debug(1, "Running %s", cmd)
+    bb.msg.debug(1, bb.msg.domain.Fetcher, "Running %s" % cmd)
 
     # redirect stderr to stdout
     stdout_handle = os.popen(cmd + " 2>&1", "r")
@@ -484,7 +451,7 @@ def try_mirrors(d, uri, mirrors, check = False, force = False):
     """
     fpath = os.path.join(data.getVar("DL_DIR", d, 1), os.path.basename(uri))
     if not check and os.access(fpath, os.R_OK) and not force:
-        logger.debug(1, "%s already exists, skipping checkout.", fpath)
+        bb.msg.debug(1, bb.msg.domain.Fetcher, "%s already exists, skipping checkout." % fpath)
         return fpath
 
     ld = d.createCopy()
@@ -494,26 +461,24 @@ def try_mirrors(d, uri, mirrors, check = False, force = False):
             try:
                 ud = FetchData(newuri, ld)
             except bb.fetch.NoMethodError:
-                logger.debug(1, "No method for %s", uri)
+                bb.msg.debug(1, bb.msg.domain.Fetcher, "No method for %s" % uri)
                 continue
 
             ud.setup_localpath(ld)
 
             try:
                 if check:
-                    found = ud.method.checkstatus(newuri, ud, ld)
-                    if found:
-                        return found
+                    ud.method.checkstatus(newuri, ud, ld)
                 else:
                     ud.method.go(newuri, ud, ld)
-                    return ud.localpath
+                return ud.localpath
             except (bb.fetch.MissingParameterError,
                     bb.fetch.FetchError,
                     bb.fetch.MD5SumError):
                 import sys
                 (type, value, traceback) = sys.exc_info()
-                logger.debug(2, "Mirror fetch failure: %s", value)
-                bb.utils.remove(ud.localpath)
+                bb.msg.debug(2, bb.msg.domain.Fetcher, "Mirror fetch failure: %s" % value)
+                removefile(ud.localpath)
                 continue
     return None
 
@@ -532,16 +497,6 @@ class FetchData(object):
         if not self.pswd and "pswd" in self.parm:
             self.pswd = self.parm["pswd"]
         self.setup = False
-
-        if "name" in self.parm:
-            self.md5_name = "%s.md5sum" % self.parm["name"]
-            self.sha256_name = "%s.sha256sum" % self.parm["name"]
-        else:
-            self.md5_name = "md5sum"
-            self.sha256_name = "sha256sum"
-        self.md5_expected = bb.data.getVarFlag("SRC_URI", self.md5_name, d)
-        self.sha256_expected = bb.data.getVarFlag("SRC_URI", self.sha256_name, d)
-
         for m in methods:
             if m.supports(url, self, d):
                 self.method = m
@@ -604,13 +559,6 @@ class Fetch(object):
         and duplicate code execution)
         """
         return url
-    def _strip_leading_slashes(self, relpath):
-        """
-        Remove leading slash as os.path.join can't cope
-        """
-        while os.path.isabs(relpath):
-            relpath = relpath[1:]
-        return relpath
 
     def setUrls(self, urls):
         self.__urls = urls
@@ -626,7 +574,7 @@ class Fetch(object):
         """
         return False
 
-    def supports_srcrev(self):
+    def suppports_srcrev(self):
         """
         The fetcher supports auto source revisions (SRCREV)
         """
@@ -655,7 +603,7 @@ class Fetch(object):
         Check the status of a URL
         Assumes localpath was called first
         """
-        logger.info("URL %s could not be checked for status since no method exists.", url)
+        bb.msg.note(1, bb.msg.domain.Fetcher, "URL %s could not be checked for status since no method exists." % url)
         return True
 
     def getSRCDate(urldata, d):
@@ -696,14 +644,14 @@ class Fetch(object):
             if not rev:
                 rev = data.getVar("SRCREV_pn-%s_%s" % (pn, ud.parm['name']), d, 1)
             if not rev:
-                rev = data.getVar("SRCREV_%s" % (ud.parm['name']), d, 1)
+                rev = data.getVar("SRCREV_%s" % (ud.parm['name']), d, 1)           
         if not rev:
             rev = data.getVar("SRCREV", d, 1)
         if rev == "INVALID":
             raise InvalidSRCREV("Please set SRCREV to a valid value")
         if not rev:
             return False
-        if rev == "SRCREVINACTION":
+        if rev is "SRCREVINACTION":
             return True
         return rev
 
@@ -730,7 +678,9 @@ class Fetch(object):
         """
         Verify the md5sum we wanted with the one we got
         """
-        wanted_sum = ud.parm.get('md5sum')
+        wanted_sum = None
+        if 'md5sum' in ud.parm:
+            wanted_sum = ud.parm['md5sum']
         if not wanted_sum:
             return True
 
@@ -755,13 +705,15 @@ class Fetch(object):
         if not hasattr(self, "_latest_revision"):
             raise ParameterError
 
-        revs = persist_data.persist('BB_URI_HEADREVS', d)
+        pd = persist_data.PersistData(d, persistent_database_connection)
         key = self.generate_revision_key(url, ud, d)
-        try:
-            return revs[key]
-        except KeyError:
-            revs[key] = rev = self._latest_revision(url, ud, d)
-            return rev
+        rev = pd.getValue("BB_URI_HEADREVS", key)
+        if rev != None:
+            return str(rev)
+
+        rev = self._latest_revision(url, ud, d)
+        pd.setValue("BB_URI_HEADREVS", key, rev)
+        return rev
 
     def sortable_revision(self, url, ud, d):
         """
@@ -770,17 +722,17 @@ class Fetch(object):
         if hasattr(self, "_sortable_revision"):
             return self._sortable_revision(url, ud, d)
 
-        localcounts = persist_data.persist('BB_URI_LOCALCOUNT', d)
+        pd = persist_data.PersistData(d, persistent_database_connection)
         key = self.generate_revision_key(url, ud, d)
 
         latest_rev = self._build_revision(url, ud, d)
-        last_rev = localcounts.get(key + '_rev')
+        last_rev = pd.getValue("BB_URI_LOCALCOUNT", key + "_rev")
         uselocalcount = bb.data.getVar("BB_LOCALCOUNT_OVERRIDE", d, True) or False
         count = None
         if uselocalcount:
             count = Fetch.localcount_internal_helper(ud, d)
         if count is None:
-            count = localcounts.get(key + '_count')
+            count = pd.getValue("BB_URI_LOCALCOUNT", key + "_count")
 
         if last_rev == latest_rev:
             return str(count + "+" + latest_rev)
@@ -796,8 +748,8 @@ class Fetch(object):
         else:
             count = str(int(count) + 1)
 
-        localcounts[key + '_rev'] = latest_rev
-        localcounts[key + '_count'] = count
+        pd.setValue("BB_URI_LOCALCOUNT", key + "_rev", latest_rev)
+        pd.setValue("BB_URI_LOCALCOUNT", key + "_count", count)
 
         return str(count + "+" + latest_rev)
 

bitbake/lib/bb/fetch/bzr.py

@@ -25,10 +25,11 @@ BitBake 'Fetch' implementation for bzr.
 
 import os
 import sys
-import logging
 import bb
 from bb import data
-from bb.fetch import Fetch, FetchError, runfetchcmd, logger
+from bb.fetch import Fetch
+from bb.fetch import FetchError
+from bb.fetch import runfetchcmd
 
 class Bzr(Fetch):
     def supports(self, url, ud, d):
@@ -37,7 +38,10 @@ class Bzr(Fetch):
     def localpath (self, url, ud, d):
 
         # Create paths to bzr checkouts
-        relpath = self._strip_leading_slashes(ud.path)
+        relpath = ud.path
+        if relpath.startswith('/'):
+            # Remove leading slash as os.path.join can't cope
+            relpath = relpath[1:]
         ud.pkgdir = os.path.join(data.expand('${BZRDIR}', d), ud.host, relpath)
 
         revision = Fetch.srcrev_internal_helper(ud, d)
@@ -61,21 +65,23 @@ class Bzr(Fetch):
 
         basecmd = data.expand('${FETCHCMD_bzr}', d)
 
-        proto =  ud.parm.get('proto', 'http')
+        proto = "http"
+        if "proto" in ud.parm:
+            proto = ud.parm["proto"]
 
         bzrroot = ud.host + ud.path
 
         options = []
 
-        if command == "revno":
+        if command is "revno":
             bzrcmd = "%s revno %s %s://%s" % (basecmd, " ".join(options), proto, bzrroot)
         else:
             if ud.revision:
                 options.append("-r %s" % ud.revision)
 
-            if command == "fetch":
+            if command is "fetch":
                 bzrcmd = "%s co %s %s://%s" % (basecmd, " ".join(options), proto, bzrroot)
-            elif command == "update":
+            elif command is "update":
                 bzrcmd = "%s pull %s --overwrite" % (basecmd, " ".join(options))
             else:
                 raise FetchError("Invalid bzr command %s" % command)
@@ -87,29 +93,22 @@ class Bzr(Fetch):
 
         if os.access(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir), '.bzr'), os.R_OK):
             bzrcmd = self._buildbzrcommand(ud, d, "update")
-            logger.debug(1, "BZR Update %s", loc)
+            bb.msg.debug(1, bb.msg.domain.Fetcher, "BZR Update %s" % loc)
             os.chdir(os.path.join (ud.pkgdir, os.path.basename(ud.path)))
             runfetchcmd(bzrcmd, d)
         else:
-            bb.utils.remove(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir)), True)
+            os.system("rm -rf %s" % os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir)))
             bzrcmd = self._buildbzrcommand(ud, d, "fetch")
-            logger.debug(1, "BZR Checkout %s", loc)
-            bb.utils.mkdirhier(ud.pkgdir)
+            bb.msg.debug(1, bb.msg.domain.Fetcher, "BZR Checkout %s" % loc)
+            bb.mkdirhier(ud.pkgdir)
             os.chdir(ud.pkgdir)
-            logger.debug(1, "Running %s", bzrcmd)
+            bb.msg.debug(1, bb.msg.domain.Fetcher, "Running %s" % bzrcmd)
             runfetchcmd(bzrcmd, d)
 
         os.chdir(ud.pkgdir)
-
-        scmdata = ud.parm.get("scmdata", "")
-        if scmdata == "keep":
-            tar_flags = ""
-        else:
-            tar_flags = "--exclude '.bzr' --exclude '.bzrtags'"
-
         # tar them up to a defined filename
         try:
-            runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.basename(ud.pkgdir)), d)
+            runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.basename(ud.pkgdir)), d)
         except:
             t, v, tb = sys.exc_info()
             try:
@@ -118,7 +117,7 @@ class Bzr(Fetch):
                 pass
             raise t, v, tb
 
-    def supports_srcrev(self):
+    def suppports_srcrev(self):
         return True
 
     def _revision_key(self, url, ud, d):
@@ -131,7 +130,7 @@ class Bzr(Fetch):
         """
         Return the latest upstream revision number
         """
-        logger.debug(2, "BZR fetcher hitting network for %s", url)
+        bb.msg.debug(2, bb.msg.domain.Fetcher, "BZR fetcher hitting network for %s" % url)
 
         output = runfetchcmd(self._buildbzrcommand(ud, d, "revno"), d, True)
 

bitbake/lib/bb/fetch/cvs.py

@@ -27,10 +27,11 @@ BitBake build tools.
 #
 
 import os
-import logging
 import bb
 from   bb import data
-from bb.fetch import Fetch, FetchError, MissingParameterError, logger
+from   bb.fetch import Fetch
+from   bb.fetch import FetchError
+from   bb.fetch import MissingParameterError
 
 class Cvs(Fetch):
     """
@@ -47,7 +48,9 @@ class Cvs(Fetch):
             raise MissingParameterError("cvs method needs a 'module' parameter")
         ud.module = ud.parm["module"]
 
-        ud.tag = ud.parm.get('tag', "")
+        ud.tag = ""
+        if 'tag' in ud.parm:
+            ud.tag = ud.parm['tag']
 
         # Override the default date in certain cases
         if 'date' in ud.parm:
@@ -74,9 +77,17 @@ class Cvs(Fetch):
 
     def go(self, loc, ud, d):
 
-        method = ud.parm.get('method', 'pserver')
-        localdir = ud.parm.get('localdir', ud.module)
-        cvs_port = ud.parm.get('port', '')
+        method = "pserver"
+        if "method" in ud.parm:
+            method = ud.parm["method"]
+
+        localdir = ud.module
+        if "localdir" in ud.parm:
+            localdir = ud.parm["localdir"]
+
+        cvs_port = ""
+        if "port" in ud.parm:
+            cvs_port = ud.parm["port"]
 
         cvs_rsh = None
         if method == "ext":
@@ -125,21 +136,21 @@ class Cvs(Fetch):
             cvsupdatecmd = "CVS_RSH=\"%s\" %s" % (cvs_rsh, cvsupdatecmd)
 
         # create module directory
-        logger.debug(2, "Fetch: checking for module directory")
+        bb.msg.debug(2, bb.msg.domain.Fetcher, "Fetch: checking for module directory")
         pkg = data.expand('${PN}', d)
         pkgdir = os.path.join(data.expand('${CVSDIR}', localdata), pkg)
         moddir = os.path.join(pkgdir, localdir)
         if os.access(os.path.join(moddir, 'CVS'), os.R_OK):
-            logger.info("Update " + loc)
+            bb.msg.note(1, bb.msg.domain.Fetcher, "Update " + loc)
             # update sources there
             os.chdir(moddir)
             myret = os.system(cvsupdatecmd)
         else:
-            logger.info("Fetch " + loc)
+            bb.msg.note(1, bb.msg.domain.Fetcher, "Fetch " + loc)
             # check out sources there
-            bb.utils.mkdirhier(pkgdir)
+            bb.mkdirhier(pkgdir)
             os.chdir(pkgdir)
-            logger.debug(1, "Running %s", cvscmd)
+            bb.msg.debug(1, bb.msg.domain.Fetcher, "Running %s" % cvscmd)
             myret = os.system(cvscmd)
 
         if myret != 0 or not os.access(moddir, os.R_OK):
@@ -149,20 +160,14 @@ class Cvs(Fetch):
                 pass
             raise FetchError(ud.module)
 
-        scmdata = ud.parm.get("scmdata", "")
-        if scmdata == "keep":
-            tar_flags = ""
-        else:
-            tar_flags = "--exclude 'CVS'"
-
         # tar them up to a defined filename
         if 'fullpath' in ud.parm:
             os.chdir(pkgdir)
-            myret = os.system("tar %s -czf %s %s" % (tar_flags, ud.localpath, localdir))
+            myret = os.system("tar -czf %s %s" % (ud.localpath, localdir))
         else:
             os.chdir(moddir)
             os.chdir('..')
-            myret = os.system("tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.basename(moddir)))
+            myret = os.system("tar -czf %s %s" % (ud.localpath, os.path.basename(moddir)))
 
         if myret != 0:
             try:

bitbake/lib/bb/fetch/git.py

@@ -22,11 +22,9 @@ BitBake 'Fetch' git implementation
 
 import os
 import bb
-import bb.persist_data
 from   bb    import data
 from   bb.fetch import Fetch
 from   bb.fetch import runfetchcmd
-from   bb.fetch import logger
 
 class Git(Fetch):
     """Class to fetch a module or modules from git repositories"""
@@ -118,7 +116,6 @@ class Git(Fetch):
 
         repofile = os.path.join(data.getVar("DL_DIR", d, 1), ud.mirrortarball)
 
-
         coname = '%s' % (ud.tag)
         codir = os.path.join(ud.clonedir, coname)
 
@@ -131,7 +128,7 @@ class Git(Fetch):
 
         # If the checkout doesn't exist and the mirror tarball does, extract it
         if not os.path.exists(ud.clonedir) and os.path.exists(repofile):
-            bb.utils.mkdirhier(ud.clonedir)
+            bb.mkdirhier(ud.clonedir)
             os.chdir(ud.clonedir)
             runfetchcmd("tar -xzf %s" % (repofile), d)
 
@@ -156,7 +153,7 @@ class Git(Fetch):
         os.chdir(ud.clonedir)
         mirror_tarballs = data.getVar("BB_GENERATE_MIRROR_TARBALLS", d, True)
         if mirror_tarballs != "0" or 'fullclone' in ud.parm:
-            logger.info("Creating tarball of git repository")
+            bb.msg.note(1, bb.msg.domain.Fetcher, "Creating tarball of git repository")
             runfetchcmd("tar -czf %s %s" % (repofile, os.path.join(".", ".git", "*") ), d)
 
         if 'fullclone' in ud.parm:
@@ -182,25 +179,19 @@ class Git(Fetch):
             readpathspec = ""
             coprefix = os.path.join(codir, "git", "")
 
-        scmdata = ud.parm.get("scmdata", "")
-        if scmdata == "keep":
-            runfetchcmd("%s clone -n %s %s" % (ud.basecmd, ud.clonedir, coprefix), d)
-            os.chdir(coprefix)
-            runfetchcmd("%s checkout -q -f %s%s" % (ud.basecmd, ud.tag, readpathspec), d)
-        else:
-            bb.utils.mkdirhier(codir)
-            os.chdir(ud.clonedir)
-            runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.tag, readpathspec), d)
-            runfetchcmd("%s checkout-index -q -f --prefix=%s -a" % (ud.basecmd, coprefix), d)
+        bb.mkdirhier(codir)
+        os.chdir(ud.clonedir)
+        runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.tag, readpathspec), d)
+        runfetchcmd("%s checkout-index -q -f --prefix=%s -a" % (ud.basecmd, coprefix), d)
 
         os.chdir(codir)
-        logger.info("Creating tarball of git checkout")
+        bb.msg.note(1, bb.msg.domain.Fetcher, "Creating tarball of git checkout")
         runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.join(".", "*") ), d)
 
         os.chdir(ud.clonedir)
         bb.utils.prunedir(codir)
 
-    def supports_srcrev(self):
+    def suppports_srcrev(self):
         return True
 
     def _contains_ref(self, tag, d):
@@ -208,19 +199,11 @@ class Git(Fetch):
         output = runfetchcmd("%s log --pretty=oneline -n 1 %s -- 2> /dev/null | wc -l" % (basecmd, tag), d, quiet=True)
         return output.split()[0] != "0"
 
-    def _revision_key(self, url, ud, d, branch=False):
+    def _revision_key(self, url, ud, d):
         """
         Return a unique key for the url
         """
-        key = 'git:' + ud.host + ud.path.replace('/', '.')
-        if branch:
-            return key + ud.branch
-        else:
-            return key
-
-    def generate_revision_key(self, url, ud, d, branch=False):
-        key = self._revision_key(url, ud, d, branch)
-        return "%s-%s" % (key, bb.data.getVar("PN", d, True) or "")
+        return "git:" + ud.host + ud.path.replace('/', '.') + ud.branch
 
     def _latest_revision(self, url, ud, d):
         """
@@ -238,74 +221,6 @@ class Git(Fetch):
             raise bb.fetch.FetchError("Fetch command %s gave empty output\n" % (cmd))
         return output.split()[0]
 
-    def latest_revision(self, url, ud, d):
-        """
-        Look in the cache for the latest revision, if not present ask the SCM.
-        """
-        revs = bb.persist_data.persist('BB_URI_HEADREVS', d)
-
-        key = self.generate_revision_key(url, ud, d, branch=True)
-
-        try:
-            return revs[key]
-        except KeyError:
-            # Compatibility with old key format, no branch included
-            oldkey = self.generate_revision_key(url, ud, d, branch=False)
-            try:
-                rev = revs[oldkey]
-            except KeyError:
-                rev = self._latest_revision(url, ud, d)
-            else:
-                del revs[oldkey]
-            revs[key] = rev
-            return rev
-
-    def sortable_revision(self, url, ud, d):
-        """
-
-        """
-        localcounts = bb.persist_data.persist('BB_URI_LOCALCOUNT', d)
-        key = self.generate_revision_key(url, ud, d, branch=True)
-        oldkey = self.generate_revision_key(url, ud, d, branch=False)
-
-        latest_rev = self._build_revision(url, ud, d)
-        last_rev = localcounts.get(key + '_rev')
-        if last_rev is None:
-            last_rev = localcounts.get(oldkey + '_rev')
-            if last_rev is not None:
-                del localcounts[oldkey + '_rev']
-                localcounts[key + '_rev'] = last_rev
-
-        uselocalcount = bb.data.getVar("BB_LOCALCOUNT_OVERRIDE", d, True) or False
-        count = None
-        if uselocalcount:
-            count = Fetch.localcount_internal_helper(ud, d)
-        if count is None:
-            count = localcounts.get(key + '_count')
-        if count is None:
-            count = localcounts.get(oldkey + '_count')
-            if count is not None:
-                del localcounts[oldkey + '_count']
-                localcounts[key + '_count'] = count
-
-        if last_rev == latest_rev:
-            return str(count + "+" + latest_rev)
-
-        buildindex_provided = hasattr(self, "_sortable_buildindex")
-        if buildindex_provided:
-            count = self._sortable_buildindex(url, ud, d, latest_rev)
-        if count is None:
-            count = "0"
-        elif uselocalcount or buildindex_provided:
-            count = str(count)
-        else:
-            count = str(int(count) + 1)
-
-        localcounts[key + '_rev'] = latest_rev
-        localcounts[key + '_count'] = count
-
-        return str(count + "+" + latest_rev)
-
     def _build_revision(self, url, ud, d):
         return ud.tag
 
@@ -323,7 +238,7 @@ class Git(Fetch):
             print("no repo")
             self.go(None, ud, d)
             if not os.path.exists(ud.clonedir):
-                logger.error("GIT repository for %s doesn't exist in %s, cannot get sortable buildnumber, using old value", url, ud.clonedir)
+                bb.msg.error(bb.msg.domain.Fetcher, "GIT repository for %s doesn't exist in %s, cannot get sortable buildnumber, using old value" % (url, ud.clonedir))
                 return None
 
 
@@ -335,5 +250,5 @@ class Git(Fetch):
         os.chdir(cwd)
 
         buildindex = "%s" % output.split()[0]
-        logger.debug(1, "GIT repository for %s in %s is returning %s revisions in rev-list before %s", url, ud.clonedir, buildindex, rev)
+        bb.msg.debug(1, bb.msg.domain.Fetcher, "GIT repository for %s in %s is returning %s revisions in rev-list before %s" % (url, ud.clonedir, buildindex, rev))
         return buildindex

bitbake/lib/bb/fetch/hg.py

@@ -26,27 +26,21 @@ BitBake 'Fetch' implementation for mercurial DRCS (hg).
 
 import os
 import sys
-import logging
 import bb
 from bb import data
 from bb.fetch import Fetch
 from bb.fetch import FetchError
 from bb.fetch import MissingParameterError
 from bb.fetch import runfetchcmd
-from bb.fetch import logger
 
 class Hg(Fetch):
-    """Class to fetch from mercurial repositories"""
+    """Class to fetch a from mercurial repositories"""
     def supports(self, url, ud, d):
         """
         Check to see if a given url can be fetched with mercurial.
         """
         return ud.type in ['hg']
 
-    def forcefetch(self, url, ud, d):
-        revTag = ud.parm.get('rev', 'tip')
-        return revTag == "tip"
-
     def localpath(self, url, ud, d):
         if not "module" in ud.parm:
             raise MissingParameterError("hg method needs a 'module' parameter")
@@ -54,7 +48,10 @@ class Hg(Fetch):
         ud.module = ud.parm["module"]
 
         # Create paths to mercurial checkouts
-        relpath = self._strip_leading_slashes(ud.path)
+        relpath = ud.path
+        if relpath.startswith('/'):
+            # Remove leading slash as os.path.join can't cope
+            relpath = relpath[1:]
         ud.pkgdir = os.path.join(data.expand('${HGDIR}', d), ud.host, relpath)
         ud.moddir = os.path.join(ud.pkgdir, ud.module)
 
@@ -81,7 +78,9 @@ class Hg(Fetch):
 
         basecmd = data.expand('${FETCHCMD_hg}', d)
 
-        proto = ud.parm.get('proto', 'http')
+        proto = "http"
+        if "proto" in ud.parm:
+            proto = ud.parm["proto"]
 
         host = ud.host
         if proto == "file":
@@ -117,41 +116,34 @@ class Hg(Fetch):
     def go(self, loc, ud, d):
         """Fetch url"""
 
-        logger.debug(2, "Fetch: checking for module directory '" + ud.moddir + "'")
+        bb.msg.debug(2, bb.msg.domain.Fetcher, "Fetch: checking for module directory '" + ud.moddir + "'")
 
         if os.access(os.path.join(ud.moddir, '.hg'), os.R_OK):
             updatecmd = self._buildhgcommand(ud, d, "pull")
-            logger.info("Update " + loc)
+            bb.msg.note(1, bb.msg.domain.Fetcher, "Update " + loc)
             # update sources there
             os.chdir(ud.moddir)
-            logger.debug(1, "Running %s", updatecmd)
+            bb.msg.debug(1, bb.msg.domain.Fetcher, "Running %s" % updatecmd)
             runfetchcmd(updatecmd, d)
 
         else:
             fetchcmd = self._buildhgcommand(ud, d, "fetch")
-            logger.info("Fetch " + loc)
+            bb.msg.note(1, bb.msg.domain.Fetcher, "Fetch " + loc)
             # check out sources there
-            bb.utils.mkdirhier(ud.pkgdir)
+            bb.mkdirhier(ud.pkgdir)
             os.chdir(ud.pkgdir)
-            logger.debug(1, "Running %s", fetchcmd)
+            bb.msg.debug(1, bb.msg.domain.Fetcher, "Running %s" % fetchcmd)
             runfetchcmd(fetchcmd, d)
 
         # Even when we clone (fetch), we still need to update as hg's clone
         # won't checkout the specified revision if its on a branch
         updatecmd = self._buildhgcommand(ud, d, "update")
-        os.chdir(ud.moddir)
-        logger.debug(1, "Running %s", updatecmd)
+        bb.msg.debug(1, bb.msg.domain.Fetcher, "Running %s" % updatecmd)
         runfetchcmd(updatecmd, d)
 
-        scmdata = ud.parm.get("scmdata", "")
-        if scmdata == "keep":
-            tar_flags = ""
-        else:
-            tar_flags = "--exclude '.hg' --exclude '.hgrags'"
-
         os.chdir(ud.pkgdir)
         try:
-            runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.module), d)
+            runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d)
         except:
             t, v, tb = sys.exc_info()
             try:
@@ -160,7 +152,7 @@ class Hg(Fetch):
                 pass
             raise t, v, tb
 
-    def supports_srcrev(self):
+    def suppports_srcrev(self):
         return True
 
     def _latest_revision(self, url, ud, d):

bitbake/lib/bb/fetch/local.py

@@ -66,7 +66,7 @@ class Local(Fetch):
         Check the status of the url
         """
         if urldata.localpath.find("*") != -1:
-            logger.info("URL %s looks like a glob and was therefore not checked.", url)
+            bb.msg.note(1, bb.msg.domain.Fetcher, "URL %s looks like a glob and was therefore not checked." % url)
             return True
         if os.path.exists(urldata.localpath):
             return True

bitbake/lib/bb/fetch/osc.py

@@ -8,10 +8,8 @@ Based on the svn "Fetch" implementation.
 
 import  os
 import  sys
-import logging
 import  bb
 from    bb       import data
-from    bb       import utils
 from    bb.fetch import Fetch
 from    bb.fetch import FetchError
 from    bb.fetch import MissingParameterError
@@ -34,7 +32,10 @@ class Osc(Fetch):
         ud.module = ud.parm["module"]
 
         # Create paths to osc checkouts
-        relpath = self._strip_leading_slashes(ud.path)
+        relpath = ud.path
+        if relpath.startswith('/'):
+            # Remove leading slash as os.path.join can't cope
+            relpath = relpath[1:]
         ud.pkgdir = os.path.join(data.expand('${OSCDIR}', d), ud.host)
         ud.moddir = os.path.join(ud.pkgdir, relpath, ud.module)
 
@@ -60,7 +61,9 @@ class Osc(Fetch):
 
         basecmd = data.expand('${FETCHCMD_osc}', d)
 
-        proto = ud.parm.get('proto', 'ocs')
+        proto = "ocs"
+        if "proto" in ud.parm:
+            proto = ud.parm["proto"]
 
         options = []
 
@@ -69,7 +72,10 @@ class Osc(Fetch):
         if ud.revision:
             options.append("-r %s" % ud.revision)
 
-        coroot = self._strip_leading_slashes(ud.path)
+        coroot = ud.path
+        if coroot.startswith('/'):
+            # Remove leading slash as os.path.join can't cope
+            coroot= coroot[1:]
 
         if command is "fetch":
             osccmd = "%s %s co %s/%s %s" % (basecmd, config, coroot, ud.module, " ".join(options))
@@ -85,22 +91,22 @@ class Osc(Fetch):
         Fetch url
         """
 
-        logger.debug(2, "Fetch: checking for module directory '" + ud.moddir + "'")
+        bb.msg.debug(2, bb.msg.domain.Fetcher, "Fetch: checking for module directory '" + ud.moddir + "'")
 
         if os.access(os.path.join(data.expand('${OSCDIR}', d), ud.path, ud.module), os.R_OK):
             oscupdatecmd = self._buildosccommand(ud, d, "update")
-            logger.info("Update "+ loc)
+            bb.msg.note(1, bb.msg.domain.Fetcher, "Update "+ loc)
             # update sources there
             os.chdir(ud.moddir)
-            logger.debug(1, "Running %s", oscupdatecmd)
+            bb.msg.debug(1, bb.msg.domain.Fetcher, "Running %s" % oscupdatecmd)
             runfetchcmd(oscupdatecmd, d)
         else:
             oscfetchcmd = self._buildosccommand(ud, d, "fetch")
-            logger.info("Fetch " + loc)
+            bb.msg.note(1, bb.msg.domain.Fetcher, "Fetch " + loc)
             # check out sources there
-            bb.utils.mkdirhier(ud.pkgdir)
+            bb.mkdirhier(ud.pkgdir)
             os.chdir(ud.pkgdir)
-            logger.debug(1, "Running %s", oscfetchcmd)
+            bb.msg.debug(1, bb.msg.domain.Fetcher, "Running %s" % oscfetchcmd)
             runfetchcmd(oscfetchcmd, d)
 
         os.chdir(os.path.join(ud.pkgdir + ud.path))
@@ -123,8 +129,9 @@ class Osc(Fetch):
         Generate a .oscrc to be used for this run.
         """
 
-        config_path = os.path.join(data.expand('${OSCDIR}', d), "oscrc")
-        bb.utils.remove(config_path)
+        config_path = "%s/oscrc" % data.expand('${OSCDIR}', d)
+        if (os.path.exists(config_path)):
+            os.remove(config_path)
 
         f = open(config_path, 'w')
         f.write("[general]\n")

bitbake/lib/bb/fetch/perforce.py

@@ -27,12 +27,10 @@ BitBake build tools.
 
 from future_builtins import zip
 import os
-import logging
 import bb
 from   bb import data
 from   bb.fetch import Fetch
 from   bb.fetch import FetchError
-from   bb.fetch import logger
 
 class Perforce(Fetch):
     def supports(self, url, ud, d):
@@ -88,10 +86,10 @@ class Perforce(Fetch):
             depot += "@%s" % (p4date)
 
         p4cmd = data.getVar('FETCHCOMMAND_p4', d, 1)
-        logger.debug(1, "Running %s%s changes -m 1 %s", p4cmd, p4opt, depot)
+        bb.msg.debug(1, bb.msg.domain.Fetcher, "Running %s%s changes -m 1 %s" % (p4cmd, p4opt, depot))
         p4file = os.popen("%s%s changes -m 1 %s" % (p4cmd, p4opt, depot))
         cset = p4file.readline().strip()
-        logger.debug(1, "READ %s", cset)
+        bb.msg.debug(1, bb.msg.domain.Fetcher, "READ %s" % (cset))
         if not cset:
             return -1
 
@@ -113,7 +111,8 @@ class Perforce(Fetch):
         if which != -1:
             base = path[:which]
 
-        base = self._strip_leading_slashes(base)
+        if base[0] == "/":
+            base = base[1:]
 
         cset = Perforce.getcset(d, path, host, user, pswd, parm)
 
@@ -133,7 +132,10 @@ class Perforce(Fetch):
         else:
             path = depot
 
-        module = parm.get('module', os.path.basename(path))
+        if "module" in parm:
+            module = parm["module"]
+        else:
+            module = os.path.basename(path)
 
         localdata = data.createCopy(d)
         data.setVar('OVERRIDES', "p4:%s" % data.getVar('OVERRIDES', localdata), localdata)
@@ -153,13 +155,13 @@ class Perforce(Fetch):
         p4cmd = data.getVar('FETCHCOMMAND', localdata, 1)
 
         # create temp directory
-        logger.debug(2, "Fetch: creating temporary directory")
-        bb.utils.mkdirhier(data.expand('${WORKDIR}', localdata))
+        bb.msg.debug(2, bb.msg.domain.Fetcher, "Fetch: creating temporary directory")
+        bb.mkdirhier(data.expand('${WORKDIR}', localdata))
         data.setVar('TMPBASE', data.expand('${WORKDIR}/oep4.XXXXXX', localdata), localdata)
         tmppipe = os.popen(data.getVar('MKTEMPDIRCMD', localdata, 1) or "false")
         tmpfile = tmppipe.readline().strip()
         if not tmpfile:
-            logger.error("Fetch: unable to create temporary directory.. make sure 'mktemp' is in the PATH.")
+            bb.msg.error(bb.msg.domain.Fetcher, "Fetch: unable to create temporary directory.. make sure 'mktemp' is in the PATH.")
             raise FetchError(module)
 
         if "label" in parm:
@@ -169,12 +171,12 @@ class Perforce(Fetch):
             depot = "%s@%s" % (depot, cset)
 
         os.chdir(tmpfile)
-        logger.info("Fetch " + loc)
-        logger.info("%s%s files %s", p4cmd, p4opt, depot)
+        bb.msg.note(1, bb.msg.domain.Fetcher, "Fetch " + loc)
+        bb.msg.note(1, bb.msg.domain.Fetcher, "%s%s files %s" % (p4cmd, p4opt, depot))
         p4file = os.popen("%s%s files %s" % (p4cmd, p4opt, depot))
 
         if not p4file:
-            logger.error("Fetch: unable to get the P4 files from %s", depot)
+            bb.msg.error(bb.msg.domain.Fetcher, "Fetch: unable to get the P4 files from %s" % (depot))
             raise FetchError(module)
 
         count = 0
@@ -192,7 +194,7 @@ class Perforce(Fetch):
             count = count + 1
 
         if count == 0:
-            logger.error("Fetch:  No files gathered from the P4 fetch")
+            bb.msg.error(bb.msg.domain.Fetcher, "Fetch:  No files gathered from the P4 fetch")
             raise FetchError(module)
 
         myret = os.system("tar -czf %s %s" % (ud.localpath, module))
@@ -203,4 +205,4 @@ class Perforce(Fetch):
                 pass
             raise FetchError(module)
         # cleanup
-        bb.utils.prunedir(tmpfile)
+        os.system('rm -rf %s' % tmpfile)

bitbake/lib/bb/fetch/repo.py

@@ -45,11 +45,24 @@ class Repo(Fetch):
         "master".
         """
 
-        ud.proto = ud.parm.get('protocol', 'git')
-        ud.branch = ud.parm.get('branch', 'master')
-        ud.manifest = ud.parm.get('manifest', 'default.xml')
-        if not ud.manifest.endswith('.xml'):
-            ud.manifest += '.xml'
+        if "protocol" in ud.parm:
+            ud.proto = ud.parm["protocol"]
+        else:
+            ud.proto = "git"
+
+        if "branch" in ud.parm:
+            ud.branch = ud.parm["branch"]
+        else:
+            ud.branch = "master"
+
+        if "manifest" in ud.parm:
+            manifest = ud.parm["manifest"]
+            if manifest.endswith(".xml"):
+                ud.manifest = manifest
+            else:
+                ud.manifest = manifest + ".xml"
+        else:
+            ud.manifest = "default.xml"
 
         ud.localfile = data.expand("repo_%s%s_%s_%s.tar.gz" % (ud.host, ud.path.replace("/", "."), ud.manifest, ud.branch), d)
 
@@ -59,7 +72,7 @@ class Repo(Fetch):
         """Fetch url"""
 
         if os.access(os.path.join(data.getVar("DL_DIR", d, True), ud.localfile), os.R_OK):
-            logger.debug(1, "%s already exists (or was stashed). Skipping repo init / sync.", ud.localpath)
+            bb.msg.debug(1, bb.msg.domain.Fetcher, "%s already exists (or was stashed). Skipping repo init / sync." % ud.localpath)
             return
 
         gitsrcname = "%s%s" % (ud.host, ud.path.replace("/", "."))
@@ -71,7 +84,7 @@ class Repo(Fetch):
         else:
             username = ""
 
-        bb.utils.mkdirhier(os.path.join(codir, "repo"))
+        bb.mkdirhier(os.path.join(codir, "repo"))
         os.chdir(os.path.join(codir, "repo"))
         if not os.path.exists(os.path.join(codir, "repo", ".repo")):
             runfetchcmd("repo init -m %s -b %s -u %s://%s%s%s" % (ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d)
@@ -79,16 +92,10 @@ class Repo(Fetch):
         runfetchcmd("repo sync", d)
         os.chdir(codir)
 
-        scmdata = ud.parm.get("scmdata", "")
-        if scmdata == "keep":
-            tar_flags = ""
-        else:
-            tar_flags = "--exclude '.repo' --exclude '.git'"
-
         # Create a cache
-        runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.join(".", "*") ), d)
+        runfetchcmd("tar --exclude=.repo --exclude=.git -czf %s %s" % (ud.localpath, os.path.join(".", "*") ), d)
 
-    def supports_srcrev(self):
+    def suppports_srcrev(self):
         return False
 
     def _build_revision(self, url, ud, d):

bitbake/lib/bb/fetch/svk.py

@@ -26,13 +26,11 @@ This implementation is for svk. It is based on the svn implementation
 # Based on functions from the base bb module, Copyright 2003 Holger Schurig
 
 import os
-import logging
 import bb
 from   bb import data
 from   bb.fetch import Fetch
 from   bb.fetch import FetchError
 from   bb.fetch import MissingParameterError
-from   bb.fetch import logger
 
 class Svk(Fetch):
     """Class to fetch a module or modules from svk repositories"""
@@ -48,14 +46,18 @@ class Svk(Fetch):
         else:
             ud.module = ud.parm["module"]
 
-        ud.revision = ud.parm.get('rev', "")
+        ud.revision = ""
+        if 'rev' in ud.parm:
+            ud.revision = ud.parm['rev']
 
         ud.localfile = data.expand('%s_%s_%s_%s_%s.tar.gz' % (ud.module.replace('/', '.'), ud.host, ud.path.replace('/', '.'), ud.revision, ud.date), d)
 
         return os.path.join(data.getVar("DL_DIR", d, True), ud.localfile)
 
     def forcefetch(self, url, ud, d):
-        return ud.date == "now"
+        if (ud.date == "now"):
+            return True
+        return False
 
     def go(self, loc, ud, d):
         """Fetch urls"""
@@ -70,19 +72,19 @@ class Svk(Fetch):
         # create temp directory
         localdata = data.createCopy(d)
         data.update_data(localdata)
-        logger.debug(2, "Fetch: creating temporary directory")
-        bb.utils.mkdirhier(data.expand('${WORKDIR}', localdata))
+        bb.msg.debug(2, bb.msg.domain.Fetcher, "Fetch: creating temporary directory")
+        bb.mkdirhier(data.expand('${WORKDIR}', localdata))
         data.setVar('TMPBASE', data.expand('${WORKDIR}/oesvk.XXXXXX', localdata), localdata)
         tmppipe = os.popen(data.getVar('MKTEMPDIRCMD', localdata, 1) or "false")
         tmpfile = tmppipe.readline().strip()
         if not tmpfile:
-            logger.error("Fetch: unable to create temporary directory.. make sure 'mktemp' is in the PATH.")
+            bb.msg.error(bb.msg.domain.Fetcher, "Fetch: unable to create temporary directory.. make sure 'mktemp' is in the PATH.")
             raise FetchError(ud.module)
 
         # check out sources there
         os.chdir(tmpfile)
-        logger.info("Fetch " + loc)
-        logger.debug(1, "Running %s", svkcmd)
+        bb.msg.note(1, bb.msg.domain.Fetcher, "Fetch " + loc)
+        bb.msg.debug(1, bb.msg.domain.Fetcher, "Running %s" % svkcmd)
         myret = os.system(svkcmd)
         if myret != 0:
             try:
@@ -101,4 +103,4 @@ class Svk(Fetch):
                 pass
             raise FetchError(ud.module)
         # cleanup
-        bb.utils.prunedir(tmpfile)
+        os.system('rm -rf %s' % tmpfile)

bitbake/lib/bb/fetch/svn.py

@@ -25,14 +25,12 @@ BitBake 'Fetch' implementation for svn.
 
 import os
 import sys
-import logging
 import bb
 from   bb import data
 from   bb.fetch import Fetch
 from   bb.fetch import FetchError
 from   bb.fetch import MissingParameterError
 from   bb.fetch import runfetchcmd
-from   bb.fetch import logger
 
 class Svn(Fetch):
     """Class to fetch a module or modules from svn repositories"""
@@ -49,7 +47,10 @@ class Svn(Fetch):
         ud.module = ud.parm["module"]
 
         # Create paths to svn checkouts
-        relpath = self._strip_leading_slashes(ud.path)
+        relpath = ud.path
+        if relpath.startswith('/'):
+            # Remove leading slash as os.path.join can't cope
+            relpath = relpath[1:]
         ud.pkgdir = os.path.join(data.expand('${SVNDIR}', d), ud.host, relpath)
         ud.moddir = os.path.join(ud.pkgdir, ud.module)
 
@@ -91,7 +92,9 @@ class Svn(Fetch):
 
         basecmd = data.expand('${FETCHCMD_svn}', d)
 
-        proto = ud.parm.get('proto', 'svn')
+        proto = "svn"
+        if "proto" in ud.parm:
+            proto = ud.parm["proto"]
 
         svn_rsh = None
         if proto == "svn+ssh" and "rsh" in ud.parm:
@@ -133,34 +136,28 @@ class Svn(Fetch):
     def go(self, loc, ud, d):
         """Fetch url"""
 
-        logger.debug(2, "Fetch: checking for module directory '" + ud.moddir + "'")
+        bb.msg.debug(2, bb.msg.domain.Fetcher, "Fetch: checking for module directory '" + ud.moddir + "'")
 
         if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK):
             svnupdatecmd = self._buildsvncommand(ud, d, "update")
-            logger.info("Update " + loc)
+            bb.msg.note(1, bb.msg.domain.Fetcher, "Update " + loc)
             # update sources there
             os.chdir(ud.moddir)
-            logger.debug(1, "Running %s", svnupdatecmd)
+            bb.msg.debug(1, bb.msg.domain.Fetcher, "Running %s" % svnupdatecmd)
             runfetchcmd(svnupdatecmd, d)
         else:
             svnfetchcmd = self._buildsvncommand(ud, d, "fetch")
-            logger.info("Fetch " + loc)
+            bb.msg.note(1, bb.msg.domain.Fetcher, "Fetch " + loc)
             # check out sources there
-            bb.utils.mkdirhier(ud.pkgdir)
+            bb.mkdirhier(ud.pkgdir)
             os.chdir(ud.pkgdir)
-            logger.debug(1, "Running %s", svnfetchcmd)
+            bb.msg.debug(1, bb.msg.domain.Fetcher, "Running %s" % svnfetchcmd)
             runfetchcmd(svnfetchcmd, d)
 
-        scmdata = ud.parm.get("scmdata", "")
-        if scmdata == "keep":
-            tar_flags = ""
-        else:
-            tar_flags = "--exclude '.svn'"
-
         os.chdir(ud.pkgdir)
         # tar them up to a defined filename
         try:
-            runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.module), d)
+            runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d)
         except:
             t, v, tb = sys.exc_info()
             try:
@@ -169,7 +166,7 @@ class Svn(Fetch):
                 pass
             raise t, v, tb
 
-    def supports_srcrev(self):
+    def suppports_srcrev(self):
         return True
 
     def _revision_key(self, url, ud, d):
@@ -182,7 +179,7 @@ class Svn(Fetch):
         """
         Return the latest upstream revision number
         """
-        logger.debug(2, "SVN fetcher hitting network for %s", url)
+        bb.msg.debug(2, bb.msg.domain.Fetcher, "SVN fetcher hitting network for %s" % url)
 
         output = runfetchcmd("LANG=C LC_ALL=C " + self._buildsvncommand(ud, d, "info"), d, True)
 

bitbake/lib/bb/fetch/wget.py

@@ -26,11 +26,12 @@ BitBake build tools.
 # Based on functions from the base bb module, Copyright 2003 Holger Schurig
 
 import os
-import logging
 import bb
-import urllib
 from   bb import data
-from   bb.fetch import Fetch, FetchError, encodeurl, decodeurl, logger, runfetchcmd
+from   bb.fetch import Fetch
+from   bb.fetch import FetchError
+from   bb.fetch import encodeurl, decodeurl
+from   bb.fetch import runfetchcmd
 
 class Wget(Fetch):
     """Class to fetch urls via 'wget'"""
@@ -44,7 +45,7 @@ class Wget(Fetch):
 
         url = encodeurl([ud.type, ud.host, ud.path, ud.user, ud.pswd, {}])
         ud.basename = os.path.basename(ud.path)
-        ud.localfile = data.expand(urllib.unquote(ud.basename), d)
+        ud.localfile = data.expand(os.path.basename(url), d)
 
         return os.path.join(data.getVar("DL_DIR", d, True), ud.localfile)
 
@@ -67,14 +68,15 @@ class Wget(Fetch):
 
             fetchcmd = fetchcmd.replace("${URI}", uri.split(";")[0])
             fetchcmd = fetchcmd.replace("${FILE}", ud.basename)
-            logger.info("fetch " + uri)
-            logger.debug(2, "executing " + fetchcmd)
+
+            bb.msg.note(1, bb.msg.domain.Fetcher, "fetch " + uri)
+            bb.msg.debug(2, bb.msg.domain.Fetcher, "executing " + fetchcmd)
             runfetchcmd(fetchcmd, d)
 
             # Sanity check since wget can pretend it succeed when it didn't
             # Also, this used to happen if sourceforge sent us to the mirror page
             if not os.path.exists(ud.localpath) and not checkonly:
-                logger.debug(2, "The fetch command for %s returned success but %s doesn't exist?...", uri, ud.localpath)
+                bb.msg.debug(2, bb.msg.domain.Fetcher, "The fetch command for %s returned success but %s doesn't exist?..." % (uri, ud.localpath))
                 return False
 
             return True

bitbake/lib/bb/fetch2/bzr.py

@@ -1,143 +0,0 @@
-"""
-BitBake 'Fetch' implementation for bzr.
-
-"""
-
-# Copyright (C) 2007 Ross Burton
-# Copyright (C) 2007 Richard Purdie
-#
-#   Classes for obtaining upstream sources for the
-#   BitBake build tools.
-#   Copyright (C) 2003, 2004  Chris Larson
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-import os
-import sys
-import logging
-import bb
-from bb import data
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import FetchError
-from bb.fetch2 import runfetchcmd
-from bb.fetch2 import logger
-
-class Bzr(FetchMethod):
-    def supports(self, url, ud, d):
-        return ud.type in ['bzr']
-
-    def urldata_init(self, ud, d):
-        """
-        init bzr specific variable within url data
-        """
-        # Create paths to bzr checkouts
-        relpath = self._strip_leading_slashes(ud.path)
-        ud.pkgdir = os.path.join(data.expand('${BZRDIR}', d), ud.host, relpath)
-
-        ud.setup_revisons(d)
-
-        if not ud.revision:
-            ud.revision = self.latest_revision(ud.url, ud, d)
-
-        ud.localfile = data.expand('bzr_%s_%s_%s.tar.gz' % (ud.host, ud.path.replace('/', '.'), ud.revision), d)
-
-    def _buildbzrcommand(self, ud, d, command):
-        """
-        Build up an bzr commandline based on ud
-        command is "fetch", "update", "revno"
-        """
-
-        basecmd = data.expand('${FETCHCMD_bzr}', d)
-
-        proto =  ud.parm.get('proto', 'http')
-
-        bzrroot = ud.host + ud.path
-
-        options = []
-
-        if command == "revno":
-            bzrcmd = "%s revno %s %s://%s" % (basecmd, " ".join(options), proto, bzrroot)
-        else:
-            if ud.revision:
-                options.append("-r %s" % ud.revision)
-
-            if command == "fetch":
-                bzrcmd = "%s co %s %s://%s" % (basecmd, " ".join(options), proto, bzrroot)
-            elif command == "update":
-                bzrcmd = "%s pull %s --overwrite" % (basecmd, " ".join(options))
-            else:
-                raise FetchError("Invalid bzr command %s" % command, ud.url)
-
-        return bzrcmd
-
-    def download(self, loc, ud, d):
-        """Fetch url"""
-
-        if os.access(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir), '.bzr'), os.R_OK):
-            bzrcmd = self._buildbzrcommand(ud, d, "update")
-            logger.debug(1, "BZR Update %s", loc)
-            bb.fetch2.check_network_access(d, bzrcmd, ud.url)
-            os.chdir(os.path.join (ud.pkgdir, os.path.basename(ud.path)))
-            runfetchcmd(bzrcmd, d)
-        else:
-            bb.utils.remove(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir)), True)
-            bzrcmd = self._buildbzrcommand(ud, d, "fetch")
-            bb.fetch2.check_network_access(d, bzrcmd, ud.url)
-            logger.debug(1, "BZR Checkout %s", loc)
-            bb.utils.mkdirhier(ud.pkgdir)
-            os.chdir(ud.pkgdir)
-            logger.debug(1, "Running %s", bzrcmd)
-            runfetchcmd(bzrcmd, d)
-
-        os.chdir(ud.pkgdir)
-
-        scmdata = ud.parm.get("scmdata", "")
-        if scmdata == "keep":
-            tar_flags = ""
-        else:
-            tar_flags = "--exclude '.bzr' --exclude '.bzrtags'"
-
-        # tar them up to a defined filename
-        runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.basename(ud.pkgdir)), d, cleanup = [ud.localpath])
-
-    def supports_srcrev(self):
-        return True
-
-    def _revision_key(self, url, ud, d, name):
-        """
-        Return a unique key for the url
-        """
-        return "bzr:" + ud.pkgdir
-
-    def _latest_revision(self, url, ud, d, name):
-        """
-        Return the latest upstream revision number
-        """
-        logger.debug(2, "BZR fetcher hitting network for %s", url)
-
-        bb.fetch2.check_network_access(d, self._buildbzrcommand(ud, d, "revno"), ud.url)
-
-        output = runfetchcmd(self._buildbzrcommand(ud, d, "revno"), d, True)
-
-        return output.strip()
-
-    def _sortable_revision(self, url, ud, d):
-        """
-        Return a sortable revision number which in our case is the revision number
-        """
-
-        return self._build_revision(url, ud, d)
-
-    def _build_revision(self, url, ud, d):
-        return ud.revision

bitbake/lib/bb/fetch2/cvs.py

@@ -1,181 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-"""
-BitBake 'Fetch' implementations
-
-Classes for obtaining upstream sources for the
-BitBake build tools.
-
-"""
-
-# Copyright (C) 2003, 2004  Chris Larson
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-#
-#Based on functions from the base bb module, Copyright 2003 Holger Schurig
-#
-
-import os
-import logging
-import bb
-from   bb import data
-from bb.fetch2 import FetchMethod, FetchError, MissingParameterError, logger
-from bb.fetch2 import runfetchcmd
-
-class Cvs(FetchMethod):
-    """
-    Class to fetch a module or modules from cvs repositories
-    """
-    def supports(self, url, ud, d):
-        """
-        Check to see if a given url can be fetched with cvs.
-        """
-        return ud.type in ['cvs']
-
-    def urldata_init(self, ud, d):
-        if not "module" in ud.parm:
-            raise MissingParameterError("module", ud.url)
-        ud.module = ud.parm["module"]
-
-        ud.tag = ud.parm.get('tag', "")
-
-        # Override the default date in certain cases
-        if 'date' in ud.parm:
-            ud.date = ud.parm['date']
-        elif ud.tag:
-            ud.date = ""
-
-        norecurse = ''
-        if 'norecurse' in ud.parm:
-            norecurse = '_norecurse'
-
-        fullpath = ''
-        if 'fullpath' in ud.parm:
-            fullpath = '_fullpath'
-
-        ud.localfile = data.expand('%s_%s_%s_%s%s%s.tar.gz' % (ud.module.replace('/', '.'), ud.host, ud.tag, ud.date, norecurse, fullpath), d)
-
-    def need_update(self, url, ud, d):
-        if (ud.date == "now"):
-            return True
-        if not os.path.exists(ud.localpath):
-            return True
-        return False
-
-    def download(self, loc, ud, d):
-
-        method = ud.parm.get('method', 'pserver')
-        localdir = ud.parm.get('localdir', ud.module)
-        cvs_port = ud.parm.get('port', '')
-
-        cvs_rsh = None
-        if method == "ext":
-            if "rsh" in ud.parm:
-                cvs_rsh = ud.parm["rsh"]
-
-        if method == "dir":
-            cvsroot = ud.path
-        else:
-            cvsroot = ":" + method
-            cvsproxyhost = data.getVar('CVS_PROXY_HOST', d, True)
-            if cvsproxyhost:
-                cvsroot += ";proxy=" + cvsproxyhost
-            cvsproxyport = data.getVar('CVS_PROXY_PORT', d, True)
-            if cvsproxyport:
-                cvsroot += ";proxyport=" + cvsproxyport
-            cvsroot += ":" + ud.user
-            if ud.pswd:
-                cvsroot += ":" + ud.pswd
-            cvsroot += "@" + ud.host + ":" + cvs_port + ud.path
-
-        options = []
-        if 'norecurse' in ud.parm:
-            options.append("-l")
-        if ud.date:
-            # treat YYYYMMDDHHMM specially for CVS
-            if len(ud.date) == 12:
-                options.append("-D \"%s %s:%s UTC\"" % (ud.date[0:8], ud.date[8:10], ud.date[10:12]))
-            else:
-                options.append("-D \"%s UTC\"" % ud.date)
-        if ud.tag:
-            options.append("-r %s" % ud.tag)
-
-        localdata = data.createCopy(d)
-        data.setVar('OVERRIDES', "cvs:%s" % data.getVar('OVERRIDES', localdata), localdata)
-        data.update_data(localdata)
-
-        data.setVar('CVSROOT', cvsroot, localdata)
-        data.setVar('CVSCOOPTS', " ".join(options), localdata)
-        data.setVar('CVSMODULE', ud.module, localdata)
-        cvscmd = data.getVar('FETCHCOMMAND', localdata, True)
-        cvsupdatecmd = data.getVar('UPDATECOMMAND', localdata, True)
-
-        if cvs_rsh:
-            cvscmd = "CVS_RSH=\"%s\" %s" % (cvs_rsh, cvscmd)
-            cvsupdatecmd = "CVS_RSH=\"%s\" %s" % (cvs_rsh, cvsupdatecmd)
-
-        # create module directory
-        logger.debug(2, "Fetch: checking for module directory")
-        pkg = data.expand('${PN}', d)
-        pkgdir = os.path.join(data.expand('${CVSDIR}', localdata), pkg)
-        moddir = os.path.join(pkgdir, localdir)
-        if os.access(os.path.join(moddir, 'CVS'), os.R_OK):
-            logger.info("Update " + loc)
-            bb.fetch2.check_network_access(d, cvsupdatecmd, ud.url)
-            # update sources there
-            os.chdir(moddir)
-            cmd = cvsupdatecmd
-        else:
-            logger.info("Fetch " + loc)
-            # check out sources there
-            bb.utils.mkdirhier(pkgdir)
-            os.chdir(pkgdir)
-            logger.debug(1, "Running %s", cvscmd)
-            bb.fetch2.check_network_access(d, cvscmd, ud.url)
-            cmd = cvscmd
-
-        runfetchcmd(cmd, d, cleanup = [moddir])
-
-        if not os.access(moddir, os.R_OK):
-            raise FetchError("Directory %s was not readable despite sucessful fetch?!" % moddir, ud.url)
-
-        scmdata = ud.parm.get("scmdata", "")
-        if scmdata == "keep":
-            tar_flags = ""
-        else:
-            tar_flags = "--exclude 'CVS'"
-
-        # tar them up to a defined filename
-        if 'fullpath' in ud.parm:
-            os.chdir(pkgdir)
-            cmd = "tar %s -czf %s %s" % (tar_flags, ud.localpath, localdir)
-        else:
-            os.chdir(moddir)
-            os.chdir('..')
-            cmd = "tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.basename(moddir))
-
-        runfetchcmd(cmd, d, cleanup = [ud.localpath])
-
-    def clean(self, ud, d):
-        """ Clean CVS Files and tarballs """
-        
-        pkg = data.expand('${PN}', d)
-        localdata = data.createCopy(d)
-        data.setVar('OVERRIDES', "cvs:%s" % data.getVar('OVERRIDES', localdata), localdata)
-        data.update_data(localdata)
-        pkgdir = os.path.join(data.expand('${CVSDIR}', localdata), pkg)
-
-        bb.utils.remove(pkgdir, True)
-        bb.utils.remove(ud.localpath)
-

bitbake/lib/bb/fetch2/git.py

@@ -1,315 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-"""
-BitBake 'Fetch' git implementation
-
-git fetcher support the SRC_URI with format of:
-SRC_URI = "git://some.host/somepath;OptionA=xxx;OptionB=xxx;..."
-
-Supported SRC_URI options are:
-
-- branch
-   The git branch to retrieve from. The default is "master"
-
-   this option also support multiple branches fetching, branches
-   are seperated by comma. in multiple branches case, the name option
-   must have the same number of names to match the branches, which is
-   used to specify the SRC_REV for the branch
-   e.g:
-   SRC_URI="git://some.host/somepath;branch=branchX,branchY;name=nameX,nameY"
-   SRCREV_nameX = "xxxxxxxxxxxxxxxxxxxx"
-   SRCREV_nameY = "YYYYYYYYYYYYYYYYYYYY"
-
-- tag
-    The git tag to retrieve. The default is "master"
-
-- protocol
-   The method to use to access the repository. Common options are "git",
-   "http", "file" and "rsync". The default is "git"
-
-- rebaseable
-   rebaseable indicates that the upstream git repo may rebase in the future,
-   and current revision may disappear from upstream repo. This option will
-   reminder fetcher to preserve local cache carefully for future use.
-   The default value is "0", set rebaseable=1 for rebaseable git repo
-
-- nocheckout
-   Don't checkout source code when unpacking. set this option for the recipe
-   who has its own routine to checkout code.
-   The default is "0", set nocheckout=1 if needed.
-
-"""
-
-#Copyright (C) 2005 Richard Purdie
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-import os
-import bb
-from   bb    import data
-from   bb.fetch2 import FetchMethod
-from   bb.fetch2 import runfetchcmd
-from   bb.fetch2 import logger
-
-class Git(FetchMethod):
-    """Class to fetch a module or modules from git repositories"""
-    def init(self, d):
-        #
-        # Only enable _sortable revision if the key is set
-        #
-        if bb.data.getVar("BB_GIT_CLONE_FOR_SRCREV", d, True):
-            self._sortable_buildindex = self._sortable_buildindex_disabled
-    def supports(self, url, ud, d):
-        """
-        Check to see if a given url can be fetched with git.
-        """
-        return ud.type in ['git']
-
-    def urldata_init(self, ud, d):
-        """
-        init git specific variable within url data
-        so that the git method like latest_revision() can work
-        """
-        if 'protocol' in ud.parm:
-            ud.proto = ud.parm['protocol']
-        elif not ud.host:
-            ud.proto = 'file'
-        else:
-            ud.proto = "git"
-
-        if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'):
-            raise bb.fetch2.ParameterError("Invalid protocol type", ud.url)
-
-        ud.nocheckout = ud.parm.get("nocheckout","0") == "1"
-
-        ud.rebaseable = ud.parm.get("rebaseable","0") == "1"
-
-        branches = ud.parm.get("branch", "master").split(',')
-        if len(branches) != len(ud.names):
-            raise bb.fetch2.ParameterError("The number of name and branch parameters is not balanced", ud.url)
-        ud.branches = {}
-        for name in ud.names:
-            branch = branches[ud.names.index(name)]
-            ud.branches[name] = branch
-
-        ud.basecmd = data.getVar("FETCHCMD_git", d, True) or "git"
-
-        ud.write_tarballs = ((data.getVar("BB_GENERATE_MIRROR_TARBALLS", d, True) or "0") != "0") or ud.rebaseable
-
-        ud.setup_revisons(d)
-
-        for name in ud.names:
-            # Ensure anything that doesn't look like a sha256 checksum/revision is translated into one
-            if not ud.revisions[name] or len(ud.revisions[name]) != 40  or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]):
-                ud.branches[name] = ud.revisions[name]
-                ud.revisions[name] = self.latest_revision(ud.url, ud, d, name)
-
-        gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
-        # for rebaseable git repo, it is necessary to keep mirror tar ball
-        # per revision, so that even the revision disappears from the
-        # upstream repo in the future, the mirror will remain intact and still
-        # contains the revision
-        if ud.rebaseable:
-            for name in ud.names:
-                gitsrcname = gitsrcname + '_' + ud.revisions[name]
-        ud.mirrortarball = 'git2_%s.tar.gz' % (gitsrcname)
-        ud.fullmirror = os.path.join(data.getVar("DL_DIR", d, True), ud.mirrortarball)
-        ud.clonedir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
-
-        ud.localfile = ud.clonedir
-
-    def localpath(self, url, ud, d):
-        return ud.clonedir
-
-    def need_update(self, u, ud, d):
-        if not os.path.exists(ud.clonedir):
-            return True
-        os.chdir(ud.clonedir)
-        for name in ud.names:
-            if not self._contains_ref(ud.revisions[name], d):
-                return True
-        if ud.write_tarballs and not os.path.exists(ud.fullmirror):
-            return True
-        return False
-
-    def try_premirror(self, u, ud, d):
-        # If we don't do this, updating an existing checkout with only premirrors
-        # is not possible
-        if bb.data.getVar("BB_FETCH_PREMIRRORONLY", d, True) is not None:
-            return True
-        if os.path.exists(ud.clonedir):
-            return False
-        return True
-
-    def download(self, loc, ud, d):
-        """Fetch url"""
-
-        if ud.user:
-            username = ud.user + '@'
-        else:
-            username = ""
-
-        ud.repochanged = not os.path.exists(ud.fullmirror)
-
-        # If the checkout doesn't exist and the mirror tarball does, extract it
-        if not os.path.exists(ud.clonedir) and os.path.exists(ud.fullmirror):
-            bb.utils.mkdirhier(ud.clonedir)
-            os.chdir(ud.clonedir)
-            runfetchcmd("tar -xzf %s" % (ud.fullmirror), d)
-
-        repourl = "%s://%s%s%s" % (ud.proto, username, ud.host, ud.path)
-
-        # If the repo still doesn't exist, fallback to cloning it
-        if not os.path.exists(ud.clonedir):
-            clone_cmd = "%s clone --bare --mirror %s %s" % (ud.basecmd, repourl, ud.clonedir)
-            bb.fetch2.check_network_access(d, clone_cmd)
-            runfetchcmd(clone_cmd, d)
-
-        os.chdir(ud.clonedir)
-        # Update the checkout if needed
-        needupdate = False
-        for name in ud.names:
-            if not self._contains_ref(ud.revisions[name], d):
-                needupdate = True
-        if needupdate:
-            try: 
-                runfetchcmd("%s remote prune origin" % ud.basecmd, d) 
-                runfetchcmd("%s remote rm origin" % ud.basecmd, d) 
-            except bb.fetch2.FetchError:
-                logger.debug(1, "No Origin")
-
-            runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, repourl), d)
-            fetch_cmd = "%s fetch -f --prune %s refs/*:refs/*" % (ud.basecmd, repourl)
-            bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
-            runfetchcmd(fetch_cmd, d)
-            runfetchcmd("%s prune-packed" % ud.basecmd, d)
-            runfetchcmd("%s pack-redundant --all | xargs -r rm" % ud.basecmd, d)
-            ud.repochanged = True
-
-    def build_mirror_data(self, url, ud, d):
-        # Generate a mirror tarball if needed
-        if ud.write_tarballs and (ud.repochanged or not os.path.exists(ud.fullmirror)):
-            os.chdir(ud.clonedir)
-            logger.info("Creating tarball of git repository")
-            runfetchcmd("tar -czf %s %s" % (ud.fullmirror, os.path.join(".") ), d)
-
-    def unpack(self, ud, destdir, d):
-        """ unpack the downloaded src to destdir"""
-
-        subdir = ud.parm.get("subpath", "")
-        if subdir != "":
-            readpathspec = ":%s" % (subdir)
-            def_destsuffix = "%s/" % os.path.basename(subdir)
-        else:
-            readpathspec = ""
-            def_destsuffix = "git/"
-
-        destsuffix = ud.parm.get("destsuffix", def_destsuffix)
-        destdir = os.path.join(destdir, destsuffix)
-        if os.path.exists(destdir):
-            bb.utils.prunedir(destdir)
-
-        runfetchcmd("git clone -s -n %s/ %s" % (ud.clonedir, destdir), d)
-        if not ud.nocheckout:
-            os.chdir(destdir)
-            if subdir != "":
-                runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.revisions[ud.names[0]], readpathspec), d)
-                runfetchcmd("%s checkout-index -q -f -a" % ud.basecmd, d)
-            else:
-                runfetchcmd("%s checkout %s" % (ud.basecmd, ud.revisions[ud.names[0]]), d)
-        return True
-
-    def clean(self, ud, d):
-        """ clean the git directory """
-
-        bb.utils.remove(ud.localpath, True)
-        bb.utils.remove(ud.fullmirror)
-
-    def supports_srcrev(self):
-        return True
-
-    def _contains_ref(self, tag, d):
-        basecmd = data.getVar("FETCHCMD_git", d, True) or "git"
-        cmd = "%s log --pretty=oneline -n 1 %s -- 2> /dev/null | wc -l" % (basecmd, tag)
-        output = runfetchcmd(cmd, d, quiet=True)
-        if len(output.split()) > 1:
-            raise bb.fetch2.FetchError("The command '%s' gave output with more then 1 line unexpectedly, output: '%s'" % (cmd, output))
-        return output.split()[0] != "0"
-
-    def _revision_key(self, url, ud, d, name):
-        """
-        Return a unique key for the url
-        """
-        return "git:" + ud.host + ud.path.replace('/', '.') + ud.branches[name]
-
-    def _latest_revision(self, url, ud, d, name):
-        """
-        Compute the HEAD revision for the url
-        """
-        if ud.user:
-            username = ud.user + '@'
-        else:
-            username = ""
-
-        basecmd = data.getVar("FETCHCMD_git", d, True) or "git"
-        cmd = "%s ls-remote %s://%s%s%s %s" % \
-              (basecmd, ud.proto, username, ud.host, ud.path, ud.branches[name])
-        bb.fetch2.check_network_access(d, cmd)
-        output = runfetchcmd(cmd, d, True)
-        if not output:
-            raise bb.fetch2.FetchError("The command %s gave empty output unexpectedly" % cmd, url)
-        return output.split()[0]
-
-    def _build_revision(self, url, ud, d, name):
-        return ud.revisions[name]
-
-    def _sortable_buildindex_disabled(self, url, ud, d, rev):
-        """
-        Return a suitable buildindex for the revision specified. This is done by counting revisions
-        using "git rev-list" which may or may not work in different circumstances.
-        """
-
-        cwd = os.getcwd()
-
-        # Check if we have the rev already
-
-        if not os.path.exists(ud.clonedir):
-            logger.debug(1, "GIT repository for %s does not exist in %s.  \
-                          Downloading.", url, ud.clonedir)
-            self.download(None, ud, d)
-            if not os.path.exists(ud.clonedir):
-                logger.error("GIT repository for %s does not exist in %s after \
-                             download. Cannot get sortable buildnumber, using \
-                             old value", url, ud.clonedir)
-                return None
-
-
-        os.chdir(ud.clonedir)
-        if not self._contains_ref(rev, d):
-            self.download(None, ud, d)
-
-        output = runfetchcmd("%s rev-list %s -- 2> /dev/null | wc -l" % (ud.basecmd, rev), d, quiet=True)
-        os.chdir(cwd)
-
-        buildindex = "%s" % output.split()[0]
-        logger.debug(1, "GIT repository for %s in %s is returning %s revisions in rev-list before %s", url, ud.clonedir, buildindex, rev)
-        return buildindex
-
-    def checkstatus(self, uri, ud, d):
-        fetchcmd = "%s ls-remote %s" % (ud.basecmd, uri)
-        try:
-            runfetchcmd(fetchcmd, d, quiet=True)
-            return True
-        except FetchError:
-            return False

bitbake/lib/bb/fetch2/hg.py

@@ -1,176 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-"""
-BitBake 'Fetch' implementation for mercurial DRCS (hg).
-
-"""
-
-# Copyright (C) 2003, 2004  Chris Larson
-# Copyright (C) 2004        Marcin Juszkiewicz
-# Copyright (C) 2007        Robert Schuster
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-#
-# Based on functions from the base bb module, Copyright 2003 Holger Schurig
-
-import os
-import sys
-import logging
-import bb
-from bb import data
-from bb.fetch2 import FetchMethod
-from bb.fetch2 import FetchError
-from bb.fetch2 import MissingParameterError
-from bb.fetch2 import runfetchcmd
-from bb.fetch2 import logger
-
-class Hg(FetchMethod):
-    """Class to fetch from mercurial repositories"""
-    def supports(self, url, ud, d):
-        """
-        Check to see if a given url can be fetched with mercurial.
-        """
-        return ud.type in ['hg']
-
-    def urldata_init(self, ud, d):
-        """
-        init hg specific variable within url data
-        """
-        if not "module" in ud.parm:
-            raise MissingParameterError('module', ud.url)
-
-        ud.module = ud.parm["module"]
-
-        # Create paths to mercurial checkouts
-        relpath = self._strip_leading_slashes(ud.path)
-        ud.pkgdir = os.path.join(data.expand('${HGDIR}', d), ud.host, relpath)
-        ud.moddir = os.path.join(ud.pkgdir, ud.module)
-
-        ud.setup_revisons(d)
-
-        if 'rev' in ud.parm:
-            ud.revision = ud.parm['rev']
-        elif not ud.revision:
-            ud.revision = self.latest_revision(ud.url, ud, d)
-
-        ud.localfile = data.expand('%s_%s_%s_%s.tar.gz' % (ud.module.replace('/', '.'), ud.host, ud.path.replace('/', '.'), ud.revision), d)
-
-    def need_update(self, url, ud, d):
-        revTag = ud.parm.get('rev', 'tip')
-        if revTag == "tip":
-            return True
-        if not os.path.exists(ud.localpath):
-            return True
-        return False
-
-    def _buildhgcommand(self, ud, d, command):
-        """
-        Build up an hg commandline based on ud
-        command is "fetch", "update", "info"
-        """
-
-        basecmd = data.expand('${FETCHCMD_hg}', d)
-
-        proto = ud.parm.get('proto', 'http')
-
-        host = ud.host
-        if proto == "file":
-            host = "/"
-            ud.host = "localhost"
-
-        if not ud.user:
-            hgroot = host + ud.path
-        else:
-            hgroot = ud.user + "@" + host + ud.path
-
-        if command == "info":
-            return "%s identify -i %s://%s/%s" % (basecmd, proto, hgroot, ud.module)
-
-        options = [];
-        if ud.revision:
-            options.append("-r %s" % ud.revision)
-
-        if command == "fetch":
-            cmd = "%s clone %s %s://%s/%s %s" % (basecmd, " ".join(options), proto, hgroot, ud.module, ud.module)
-        elif command == "pull":
-            # do not pass options list; limiting pull to rev causes the local
-            # repo not to contain it and immediately following "update" command
-            # will crash
-            cmd = "%s pull" % (basecmd)
-        elif command == "update":
-            cmd = "%s update -C %s" % (basecmd, " ".join(options))
-        else:
-            raise FetchError("Invalid hg command %s" % command, ud.url)
-
-        return cmd
-
-    def download(self, loc, ud, d):
-        """Fetch url"""
-
-        logger.debug(2, "Fetch: checking for module directory '" + ud.moddir + "'")
-
-        if os.access(os.path.join(ud.moddir, '.hg'), os.R_OK):
-            updatecmd = self._buildhgcommand(ud, d, "pull")
-            logger.info("Update " + loc)
-            # update sources there
-            os.chdir(ud.moddir)
-            logger.debug(1, "Running %s", updatecmd)
-            bb.fetch2.check_network_access(d, updatecmd, ud.url)
-            runfetchcmd(updatecmd, d)
-
-        else:
-            fetchcmd = self._buildhgcommand(ud, d, "fetch")
-            logger.info("Fetch " + loc)
-            # check out sources there
-            bb.utils.mkdirhier(ud.pkgdir)
-            os.chdir(ud.pkgdir)
-            logger.debug(1, "Running %s", fetchcmd)
-            bb.fetch2.check_network_access(d, fetchcmd, ud.url)
-            runfetchcmd(fetchcmd, d)
-
-        # Even when we clone (fetch), we still need to update as hg's clone
-        # won't checkout the specified revision if its on a branch
-        updatecmd = self._buildhgcommand(ud, d, "update")
-        os.chdir(ud.moddir)
-        logger.debug(1, "Running %s", updatecmd)
-        runfetchcmd(updatecmd, d)
-
-        scmdata = ud.parm.get("scmdata", "")
-        if scmdata == "keep":
-            tar_flags = ""
-        else:
-            tar_flags = "--exclude '.hg' --exclude '.hgrags'"
-
-        os.chdir(ud.pkgdir)
-        runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.module), d, cleanup = [ud.localpath])
-
-    def supports_srcrev(self):
-        return True
-
-    def _latest_revision(self, url, ud, d, name):
-        """
-        Compute tip revision for the url
-        """
-        bb.fetch2.check_network_access(d, self._buildhgcommand(ud, d, "info"))
-        output = runfetchcmd(self._buildhgcommand(ud, d, "info"), d)
-        return output.strip()
-
-    def _build_revision(self, url, ud, d, name):
-        return ud.revision
-
-    def _revision_key(self, url, ud, d, name):
-        """
-        Return a unique key for the url
-        """
-        return "hg:" + ud.moddir

bitbake/lib/bb/fetch2/local.py

@@ -1,91 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-"""
-BitBake 'Fetch' implementations
-
-Classes for obtaining upstream sources for the
-BitBake build tools.
-
-"""
-
-# Copyright (C) 2003, 2004  Chris Larson
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-#
-# Based on functions from the base bb module, Copyright 2003 Holger Schurig
-
-import os
-import bb
-import bb.utils
-from   bb import data
-from   bb.fetch2 import FetchMethod
-
-class Local(FetchMethod):
-    def supports(self, url, urldata, d):
-        """
-        Check to see if a given url represents a local fetch.
-        """
-        return urldata.type in ['file']
-
-    def urldata_init(self, ud, d):
-        # We don't set localfile as for this fetcher the file is already local!
-        ud.basename = os.path.basename(ud.url.split("://")[1].split(";")[0])
-        return
-
-    def localpath(self, url, urldata, d):
-        """
-        Return the local filename of a given url assuming a successful fetch.
-        """
-        path = url.split("://")[1]
-        path = path.split(";")[0]
-        newpath = path
-        if path[0] != "/":
-            filespath = data.getVar('FILESPATH', d, True)
-            if filespath:
-                newpath = bb.utils.which(filespath, path)
-            if not newpath:
-                filesdir = data.getVar('FILESDIR', d, True)
-                if filesdir:
-                    newpath = os.path.join(filesdir, path)
-        if not os.path.exists(newpath) and path.find("*") == -1:
-            dldirfile = os.path.join(data.getVar("DL_DIR", d, True), os.path.basename(path))
-            return dldirfile
-        return newpath
-
-    def need_update(self, url, ud, d):
-        if url.find("*") != -1:
-            return False
-        if os.path.exists(ud.localpath):
-            return False
-        return True
-
-    def download(self, url, urldata, d):
-        """Fetch urls (no-op for Local method)"""
-        # no need to fetch local files, we'll deal with them in place.
-        return 1
-
-    def checkstatus(self, url, urldata, d):
-        """
-        Check the status of the url
-        """
-        if urldata.localpath.find("*") != -1:
-            logger.info("URL %s looks like a glob and was therefore not checked.", url)
-            return True
-        if os.path.exists(urldata.localpath):
-            return True
-        return False
-
-    def clean(self, urldata, d):
-        return
-

bitbake/lib/bb/fetch2/osc.py

@@ -1,135 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-"""
-Bitbake "Fetch" implementation for osc (Opensuse build service client).
-Based on the svn "Fetch" implementation.
-
-"""
-
-import  os
-import  sys
-import logging
-import  bb
-from    bb       import data
-from    bb.fetch2 import FetchMethod
-from    bb.fetch2 import FetchError
-from    bb.fetch2 import MissingParameterError
-from    bb.fetch2 import runfetchcmd
-
-class Osc(FetchMethod):
-    """Class to fetch a module or modules from Opensuse build server
-       repositories."""
-
-    def supports(self, url, ud, d):
-        """
-        Check to see if a given url can be fetched with osc.
-        """
-        return ud.type in ['osc']
-
-    def urldata_init(self, ud, d):
-        if not "module" in ud.parm:
-            raise MissingParameterError('module', ud.url)
-
-        ud.module = ud.parm["module"]
-
-        # Create paths to osc checkouts
-        relpath = self._strip_leading_slashes(ud.path)
-        ud.pkgdir = os.path.join(data.expand('${OSCDIR}', d), ud.host)
-        ud.moddir = os.path.join(ud.pkgdir, relpath, ud.module)
-
-        if 'rev' in ud.parm:
-            ud.revision = ud.parm['rev']
-        else:
-            pv = data.getVar("PV", d, 0)
-            rev = bb.fetch2.srcrev_internal_helper(ud, d)
-            if rev and rev != True:
-                ud.revision = rev
-            else:
-                ud.revision = ""
-
-        ud.localfile = data.expand('%s_%s_%s.tar.gz' % (ud.module.replace('/', '.'), ud.path.replace('/', '.'), ud.revision), d)
-
-    def _buildosccommand(self, ud, d, command):
-        """
-        Build up an ocs commandline based on ud
-        command is "fetch", "update", "info"
-        """
-
-        basecmd = data.expand('${FETCHCMD_osc}', d)
-
-        proto = ud.parm.get('proto', 'ocs')
-
-        options = []
-
-        config = "-c %s" % self.generate_config(ud, d)
-
-        if ud.revision:
-            options.append("-r %s" % ud.revision)
-
-        coroot = self._strip_leading_slashes(ud.path)
-
-        if command == "fetch":
-            osccmd = "%s %s co %s/%s %s" % (basecmd, config, coroot, ud.module, " ".join(options))
-        elif command == "update":
-            osccmd = "%s %s up %s" % (basecmd, config, " ".join(options))
-        else:
-            raise FetchError("Invalid osc command %s" % command, ud.url)
-
-        return osccmd
-
-    def download(self, loc, ud, d):
-        """
-        Fetch url
-        """
-
-        logger.debug(2, "Fetch: checking for module directory '" + ud.moddir + "'")
-
-        if os.access(os.path.join(data.expand('${OSCDIR}', d), ud.path, ud.module), os.R_OK):
-            oscupdatecmd = self._buildosccommand(ud, d, "update")
-            logger.info("Update "+ loc)
-            # update sources there
-            os.chdir(ud.moddir)
-            logger.debug(1, "Running %s", oscupdatecmd)
-            bb.fetch2.check_network_access(d, oscupdatecmd, ud.url)
-            runfetchcmd(oscupdatecmd, d)
-        else:
-            oscfetchcmd = self._buildosccommand(ud, d, "fetch")
-            logger.info("Fetch " + loc)
-            # check out sources there
-            bb.utils.mkdirhier(ud.pkgdir)
-            os.chdir(ud.pkgdir)
-            logger.debug(1, "Running %s", oscfetchcmd)
-            bb.fetch2.check_network_access(d, oscfetchcmd, ud.url)
-            runfetchcmd(oscfetchcmd, d)
-
-        os.chdir(os.path.join(ud.pkgdir + ud.path))
-        # tar them up to a defined filename
-        runfetchcmd("tar -czf %s %s" % (ud.localpath, ud.module), d, cleanup = [ud.localpath])
-
-    def supports_srcrev(self):
-        return False
-
-    def generate_config(self, ud, d):
-        """
-        Generate a .oscrc to be used for this run.
-        """
-
-        config_path = os.path.join(data.expand('${OSCDIR}', d), "oscrc")
-        if (os.path.exists(config_path)):
-            os.remove(config_path)
-
-        f = open(config_path, 'w')
-        f.write("[general]\n")
-        f.write("apisrv = %s\n" % ud.host)
-        f.write("scheme = http\n")
-        f.write("su-wrapper = su -c\n")
-        f.write("build-root = %s\n" % data.expand('${WORKDIR}', d))
-        f.write("urllist = http://moblin-obs.jf.intel.com:8888/build/%(project)s/%(repository)s/%(buildarch)s/:full/%(name)s.rpm\n")
-        f.write("extra-pkgs = gzip\n")
-        f.write("\n")
-        f.write("[%s]\n" % ud.host)
-        f.write("user = %s\n" % ud.parm["user"])
-        f.write("pass = %s\n" % ud.parm["pswd"])
-        f.close()
-
-        return config_path

bitbake/lib/bb/fetch2/perforce.py

@@ -1,196 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-"""
-BitBake 'Fetch' implementations
-
-Classes for obtaining upstream sources for the
-BitBake build tools.
-
-"""
-
-# Copyright (C) 2003, 2004  Chris Larson
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-#
-# Based on functions from the base bb module, Copyright 2003 Holger Schurig
-
-from future_builtins import zip
-import os
-import logging
-import bb
-from   bb import data
-from   bb.fetch2 import FetchMethod
-from   bb.fetch2 import FetchError
-from   bb.fetch2 import logger
-from   bb.fetch2 import runfetchcmd
-
-class Perforce(FetchMethod):
-    def supports(self, url, ud, d):
-        return ud.type in ['p4']
-
-    def doparse(url, d):
-        parm = {}
-        path = url.split("://")[1]
-        delim = path.find("@");
-        if delim != -1:
-            (user, pswd, host, port) = path.split('@')[0].split(":")
-            path = path.split('@')[1]
-        else:
-            (host, port) = data.getVar('P4PORT', d).split(':')
-            user = ""
-            pswd = ""
-
-        if path.find(";") != -1:
-            keys=[]
-            values=[]
-            plist = path.split(';')
-            for item in plist:
-                if item.count('='):
-                    (key, value) = item.split('=')
-                    keys.append(key)
-                    values.append(value)
-
-            parm = dict(zip(keys, values))
-        path = "//" + path.split(';')[0]
-        host += ":%s" % (port)
-        parm["cset"] = Perforce.getcset(d, path, host, user, pswd, parm)
-
-        return host, path, user, pswd, parm
-    doparse = staticmethod(doparse)
-
-    def getcset(d, depot, host, user, pswd, parm):
-        p4opt = ""
-        if "cset" in parm:
-            return parm["cset"];
-        if user:
-            p4opt += " -u %s" % (user)
-        if pswd:
-            p4opt += " -P %s" % (pswd)
-        if host:
-            p4opt += " -p %s" % (host)
-
-        p4date = data.getVar("P4DATE", d, True)
-        if "revision" in parm:
-            depot += "#%s" % (parm["revision"])
-        elif "label" in parm:
-            depot += "@%s" % (parm["label"])
-        elif p4date:
-            depot += "@%s" % (p4date)
-
-        p4cmd = data.getVar('FETCHCOMMAND_p4', d, True)
-        logger.debug(1, "Running %s%s changes -m 1 %s", p4cmd, p4opt, depot)
-        p4file = os.popen("%s%s changes -m 1 %s" % (p4cmd, p4opt, depot))
-        cset = p4file.readline().strip()
-        logger.debug(1, "READ %s", cset)
-        if not cset:
-            return -1
-
-        return cset.split(' ')[1]
-    getcset = staticmethod(getcset)
-
-    def urldata_init(self, ud, d):
-        (host, path, user, pswd, parm) = Perforce.doparse(ud.url, d)
-
-        # If a label is specified, we use that as our filename
-
-        if "label" in parm:
-            ud.localfile = "%s.tar.gz" % (parm["label"])
-            return
-
-        base = path
-        which = path.find('/...')
-        if which != -1:
-            base = path[:which]
-
-        base = self._strip_leading_slashes(base)
-
-        cset = Perforce.getcset(d, path, host, user, pswd, parm)
-
-        ud.localfile = data.expand('%s+%s+%s.tar.gz' % (host, base.replace('/', '.'), cset), d)
-
-    def download(self, loc, ud, d):
-        """
-        Fetch urls
-        """
-
-        (host, depot, user, pswd, parm) = Perforce.doparse(loc, d)
-
-        if depot.find('/...') != -1:
-            path = depot[:depot.find('/...')]
-        else:
-            path = depot
-
-        module = parm.get('module', os.path.basename(path))
-
-        localdata = data.createCopy(d)
-        data.setVar('OVERRIDES', "p4:%s" % data.getVar('OVERRIDES', localdata), localdata)
-        data.update_data(localdata)
-
-        # Get the p4 command
-        p4opt = ""
-        if user:
-            p4opt += " -u %s" % (user)
-
-        if pswd:
-            p4opt += " -P %s" % (pswd)
-
-        if host:
-            p4opt += " -p %s" % (host)
-
-        p4cmd = data.getVar('FETCHCOMMAND', localdata, True)
-
-        # create temp directory
-        logger.debug(2, "Fetch: creating temporary directory")
-        bb.utils.mkdirhier(data.expand('${WORKDIR}', localdata))
-        data.setVar('TMPBASE', data.expand('${WORKDIR}/oep4.XXXXXX', localdata), localdata)
-        tmppipe = os.popen(data.getVar('MKTEMPDIRCMD', localdata, True) or "false")
-        tmpfile = tmppipe.readline().strip()
-        if not tmpfile:
-            raise FetchError("Fetch: unable to create temporary directory.. make sure 'mktemp' is in the PATH.", loc)
-
-        if "label" in parm:
-            depot = "%s@%s" % (depot, parm["label"])
-        else:
-            cset = Perforce.getcset(d, depot, host, user, pswd, parm)
-            depot = "%s@%s" % (depot, cset)
-
-        os.chdir(tmpfile)
-        logger.info("Fetch " + loc)
-        logger.info("%s%s files %s", p4cmd, p4opt, depot)
-        p4file = os.popen("%s%s files %s" % (p4cmd, p4opt, depot))
-
-        if not p4file:
-            raise FetchError("Fetch: unable to get the P4 files from %s" % depot, loc)
-
-        count = 0
-
-        for file in p4file:
-            list = file.split()
-
-            if list[2] == "delete":
-                continue
-
-            dest = list[0][len(path)+1:]
-            where = dest.find("#")
-
-            os.system("%s%s print -o %s/%s %s" % (p4cmd, p4opt, module, dest[:where], list[0]))
-            count = count + 1
-
-        if count == 0:
-            logger.error()
-            raise FetchError("Fetch: No files gathered from the P4 fetch", loc)
-
-        runfetchcmd("tar -czf %s %s" % (ud.localpath, module), d, cleanup = [ud.localpath])
-        # cleanup
-        bb.utils.prunedir(tmpfile)

bitbake/lib/bb/fetch2/repo.py

@@ -1,98 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-"""
-BitBake "Fetch" repo (git) implementation
-
-"""
-
-# Copyright (C) 2009 Tom Rini <trini@embeddedalley.com>
-#
-# Based on git.py which is:
-#Copyright (C) 2005 Richard Purdie
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-import os
-import bb
-from   bb    import data
-from   bb.fetch2 import FetchMethod
-from   bb.fetch2 import runfetchcmd
-
-class Repo(FetchMethod):
-    """Class to fetch a module or modules from repo (git) repositories"""
-    def supports(self, url, ud, d):
-        """
-        Check to see if a given url can be fetched with repo.
-        """
-        return ud.type in ["repo"]
-
-    def urldata_init(self, ud, d):
-        """
-        We don"t care about the git rev of the manifests repository, but
-        we do care about the manifest to use.  The default is "default".
-        We also care about the branch or tag to be used.  The default is
-        "master".
-        """
-
-        ud.proto = ud.parm.get('protocol', 'git')
-        ud.branch = ud.parm.get('branch', 'master')
-        ud.manifest = ud.parm.get('manifest', 'default.xml')
-        if not ud.manifest.endswith('.xml'):
-            ud.manifest += '.xml'
-
-        ud.localfile = data.expand("repo_%s%s_%s_%s.tar.gz" % (ud.host, ud.path.replace("/", "."), ud.manifest, ud.branch), d)
-
-    def download(self, loc, ud, d):
-        """Fetch url"""
-
-        if os.access(os.path.join(data.getVar("DL_DIR", d, True), ud.localfile), os.R_OK):
-            logger.debug(1, "%s already exists (or was stashed). Skipping repo init / sync.", ud.localpath)
-            return
-
-        gitsrcname = "%s%s" % (ud.host, ud.path.replace("/", "."))
-        repodir = data.getVar("REPODIR", d, True) or os.path.join(data.getVar("DL_DIR", d, True), "repo")
-        codir = os.path.join(repodir, gitsrcname, ud.manifest)
-
-        if ud.user:
-            username = ud.user + "@"
-        else:
-            username = ""
-
-        bb.utils.mkdirhier(os.path.join(codir, "repo"))
-        os.chdir(os.path.join(codir, "repo"))
-        if not os.path.exists(os.path.join(codir, "repo", ".repo")):
-            bb.fetch2.check_network_access(d, "repo init -m %s -b %s -u %s://%s%s%s" % (ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), ud.url)
-            runfetchcmd("repo init -m %s -b %s -u %s://%s%s%s" % (ud.manifest, ud.branch, ud.proto, username, ud.host, ud.path), d)
-
-        bb.fetch2.check_network_access(d, "repo sync %s" % ud.url, ud.url)
-        runfetchcmd("repo sync", d)
-        os.chdir(codir)
-
-        scmdata = ud.parm.get("scmdata", "")
-        if scmdata == "keep":
-            tar_flags = ""
-        else:
-            tar_flags = "--exclude '.repo' --exclude '.git'"
-
-        # Create a cache
-        runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, os.path.join(".", "*") ), d)
-
-    def supports_srcrev(self):
-        return False
-
-    def _build_revision(self, url, ud, d):
-        return ud.manifest
-
-    def _want_sortable_revision(self, url, ud, d):
-        return False

bitbake/lib/bb/fetch2/ssh.py

@@ -1,120 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-'''
-BitBake 'Fetch' implementations
-
-This implementation is for Secure Shell (SSH), and attempts to comply with the
-IETF secsh internet draft:
-    http://tools.ietf.org/wg/secsh/draft-ietf-secsh-scp-sftp-ssh-uri/
-
-    Currently does not support the sftp parameters, as this uses scp
-    Also does not support the 'fingerprint' connection parameter.
-
-'''
-
-# Copyright (C) 2006  OpenedHand Ltd.
-#
-#
-# Based in part on svk.py:
-#    Copyright (C) 2006 Holger Hans Peter Freyther
-#    Based on svn.py:
-#        Copyright (C) 2003, 2004  Chris Larson
-#        Based on functions from the base bb module:
-#            Copyright 2003 Holger Schurig
-#
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-import re, os
-from   bb import data
-from   bb.fetch2 import FetchMethod
-from   bb.fetch2 import FetchError
-from   bb.fetch2 import logger
-from   bb.fetch2 import runfetchcmd
-
-
-__pattern__ = re.compile(r'''
- \s*                 # Skip leading whitespace
- ssh://              # scheme
- (                   # Optional username/password block
-  (?P<user>\S+)      # username
-  (:(?P<pass>\S+))?  # colon followed by the password (optional)
- )?
- (?P<cparam>(;[^;]+)*)?  # connection parameters block (optional)
- @
- (?P<host>\S+?)          # non-greedy match of the host
- (:(?P<port>[0-9]+))?    # colon followed by the port (optional)
- /
- (?P<path>[^;]+)         # path on the remote system, may be absolute or relative,
-                         # and may include the use of '~' to reference the remote home
-                         # directory
- (?P<sparam>(;[^;]+)*)?  # parameters block (optional)
- $
-''', re.VERBOSE)
-
-class SSH(FetchMethod):
-    '''Class to fetch a module or modules via Secure Shell'''
-
-    def supports(self, url, urldata, d):
-        return __pattern__.match(url) != None
-
-    def localpath(self, url, urldata, d):
-        m = __pattern__.match(urldata.url)
-        path = m.group('path')
-        host = m.group('host')
-        lpath = os.path.join(data.getVar('DL_DIR', d, True), host, os.path.basename(path))
-        return lpath
-
-    def download(self, url, urldata, d):
-        dldir = data.getVar('DL_DIR', d, True)
-
-        m = __pattern__.match(url)
-        path = m.group('path')
-        host = m.group('host')
-        port = m.group('port')
-        user = m.group('user')
-        password = m.group('pass')
-
-        ldir = os.path.join(dldir, host)
-        lpath = os.path.join(ldir, os.path.basename(path))
-
-        if not os.path.exists(ldir):
-            os.makedirs(ldir)
-
-        if port:
-            port = '-P %s' % port
-        else:
-            port = ''
-
-        if user:
-            fr = user
-            if password:
-                fr += ':%s' % password
-            fr += '@%s' % host
-        else:
-            fr = host
-        fr += ':%s' % path
-
-
-        import commands
-        cmd = 'scp -B -r %s %s %s/' % (
-            port,
-            commands.mkarg(fr),
-            commands.mkarg(ldir)
-        )
-
-        bb.fetch2.check_network_access(d, cmd, urldata.url)
-
-        runfetchcmd(cmd, d)
-

bitbake/lib/bb/fetch2/svk.py

@@ -1,97 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-"""
-BitBake 'Fetch' implementations
-
-This implementation is for svk. It is based on the svn implementation
-
-"""
-
-# Copyright (C) 2006 Holger Hans Peter Freyther
-# Copyright (C) 2003, 2004  Chris Larson
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-#
-# Based on functions from the base bb module, Copyright 2003 Holger Schurig
-
-import os
-import logging
-import bb
-from   bb import data
-from   bb.fetch2 import FetchMethod
-from   bb.fetch2 import FetchError
-from   bb.fetch2 import MissingParameterError
-from   bb.fetch2 import logger
-from   bb.fetch2 import runfetchcmd
-
-class Svk(FetchMethod):
-    """Class to fetch a module or modules from svk repositories"""
-    def supports(self, url, ud, d):
-        """
-        Check to see if a given url can be fetched with svk.
-        """
-        return ud.type in ['svk']
-
-    def urldata_init(self, ud, d):
-
-        if not "module" in ud.parm:
-            raise MissingParameterError('module', ud.url)
-        else:
-            ud.module = ud.parm["module"]
-
-        ud.revision = ud.parm.get('rev', "")
-
-        ud.localfile = data.expand('%s_%s_%s_%s_%s.tar.gz' % (ud.module.replace('/', '.'), ud.host, ud.path.replace('/', '.'), ud.revision, ud.date), d)
-
-    def need_update(self, url, ud, d):
-        if ud.date == "now":
-            return True
-        if not os.path.exists(ud.localpath):
-            return True
-        return False
-
-    def download(self, loc, ud, d):
-        """Fetch urls"""
-
-        svkroot = ud.host + ud.path
-
-        svkcmd = "svk co -r {%s} %s/%s" % (ud.date, svkroot, ud.module)
-
-        if ud.revision:
-            svkcmd = "svk co -r %s %s/%s" % (ud.revision, svkroot, ud.module)
-
-        # create temp directory
-        localdata = data.createCopy(d)
-        data.update_data(localdata)
-        logger.debug(2, "Fetch: creating temporary directory")
-        bb.utils.mkdirhier(data.expand('${WORKDIR}', localdata))
-        data.setVar('TMPBASE', data.expand('${WORKDIR}/oesvk.XXXXXX', localdata), localdata)
-        tmppipe = os.popen(data.getVar('MKTEMPDIRCMD', localdata, True) or "false")
-        tmpfile = tmppipe.readline().strip()
-        if not tmpfile:
-            logger.error()
-            raise FetchError("Fetch: unable to create temporary directory.. make sure 'mktemp' is in the PATH.", loc)
-
-        # check out sources there
-        os.chdir(tmpfile)
-        logger.info("Fetch " + loc)
-        logger.debug(1, "Running %s", svkcmd)
-        runfetchcmd(svkcmd, d, cleanup = [tmpfile])
-
-        os.chdir(os.path.join(tmpfile, os.path.dirname(ud.module)))
-        # tar them up to a defined filename
-        runfetchcmd("tar -czf %s %s" % (ud.localpath, os.path.basename(ud.module)), d, cleanup = [ud.localpath])
-
-        # cleanup
-        bb.utils.prunedir(tmpfile)

bitbake/lib/bb/fetch2/svn.py

@@ -1,182 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-"""
-BitBake 'Fetch' implementation for svn.
-
-"""
-
-# Copyright (C) 2003, 2004  Chris Larson
-# Copyright (C) 2004        Marcin Juszkiewicz
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-#
-# Based on functions from the base bb module, Copyright 2003 Holger Schurig
-
-import os
-import sys
-import logging
-import bb
-from   bb import data
-from   bb.fetch2 import FetchMethod
-from   bb.fetch2 import FetchError
-from   bb.fetch2 import MissingParameterError
-from   bb.fetch2 import runfetchcmd
-from   bb.fetch2 import logger
-
-class Svn(FetchMethod):
-    """Class to fetch a module or modules from svn repositories"""
-    def supports(self, url, ud, d):
-        """
-        Check to see if a given url can be fetched with svn.
-        """
-        return ud.type in ['svn']
-
-    def urldata_init(self, ud, d):
-        """
-        init svn specific variable within url data
-        """
-        if not "module" in ud.parm:
-            raise MissingParameterError('module', ud.url)
-
-        ud.module = ud.parm["module"]
-
-        # Create paths to svn checkouts
-        relpath = self._strip_leading_slashes(ud.path)
-        ud.pkgdir = os.path.join(data.expand('${SVNDIR}', d), ud.host, relpath)
-        ud.moddir = os.path.join(ud.pkgdir, ud.module)
-
-        ud.setup_revisons(d)
-
-        if 'rev' in ud.parm:
-            ud.revision = ud.parm['rev']
-
-        ud.localfile = data.expand('%s_%s_%s_%s_.tar.gz' % (ud.module.replace('/', '.'), ud.host, ud.path.replace('/', '.'), ud.revision), d)
-
-    def _buildsvncommand(self, ud, d, command):
-        """
-        Build up an svn commandline based on ud
-        command is "fetch", "update", "info"
-        """
-
-        basecmd = data.expand('${FETCHCMD_svn}', d)
-
-        proto = ud.parm.get('proto', 'svn')
-
-        svn_rsh = None
-        if proto == "svn+ssh" and "rsh" in ud.parm:
-            svn_rsh = ud.parm["rsh"]
-
-        svnroot = ud.host + ud.path
-
-        options = []
-
-        if ud.user:
-            options.append("--username %s" % ud.user)
-
-        if ud.pswd:
-            options.append("--password %s" % ud.pswd)
-
-        if command == "info":
-            svncmd = "%s info %s %s://%s/%s/" % (basecmd, " ".join(options), proto, svnroot, ud.module)
-        else:
-            suffix = ""
-            if ud.revision:
-                options.append("-r %s" % ud.revision)
-                suffix = "@%s" % (ud.revision)
-
-            if command == "fetch":
-                svncmd = "%s co %s %s://%s/%s%s %s" % (basecmd, " ".join(options), proto, svnroot, ud.module, suffix, ud.module)
-            elif command == "update":
-                svncmd = "%s update %s" % (basecmd, " ".join(options))
-            else:
-                raise FetchError("Invalid svn command %s" % command, ud.url)
-
-        if svn_rsh:
-            svncmd = "svn_RSH=\"%s\" %s" % (svn_rsh, svncmd)
-
-        return svncmd
-
-    def download(self, loc, ud, d):
-        """Fetch url"""
-
-        logger.debug(2, "Fetch: checking for module directory '" + ud.moddir + "'")
-
-        if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK):
-            svnupdatecmd = self._buildsvncommand(ud, d, "update")
-            logger.info("Update " + loc)
-            # update sources there
-            os.chdir(ud.moddir)
-            logger.debug(1, "Running %s", svnupdatecmd)
-            bb.fetch2.check_network_access(d, svnupdatecmd, ud.url)
-            runfetchcmd(svnupdatecmd, d)
-        else:
-            svnfetchcmd = self._buildsvncommand(ud, d, "fetch")
-            logger.info("Fetch " + loc)
-            # check out sources there
-            bb.utils.mkdirhier(ud.pkgdir)
-            os.chdir(ud.pkgdir)
-            logger.debug(1, "Running %s", svnfetchcmd)
-            bb.fetch2.check_network_access(d, svnfetchcmd, ud.url)
-            runfetchcmd(svnfetchcmd, d)
-
-        scmdata = ud.parm.get("scmdata", "")
-        if scmdata == "keep":
-            tar_flags = ""
-        else:
-            tar_flags = "--exclude '.svn'"
-
-        os.chdir(ud.pkgdir)
-        # tar them up to a defined filename
-        runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.module), d, cleanup = [ud.localpath])
-
-    def clean(self, ud, d):
-        """ Clean SVN specific files and dirs """
-
-        bb.utils.remove(ud.localpath)
-        bb.utils.remove(ud.moddir, True)
-        
-
-    def supports_srcrev(self):
-        return True
-
-    def _revision_key(self, url, ud, d, name):
-        """
-        Return a unique key for the url
-        """
-        return "svn:" + ud.moddir
-
-    def _latest_revision(self, url, ud, d, name):
-        """
-        Return the latest upstream revision number
-        """
-        bb.fetch2.check_network_access(d, self._buildsvncommand(ud, d, "info"))
-
-        output = runfetchcmd("LANG=C LC_ALL=C " + self._buildsvncommand(ud, d, "info"), d, True)
-
-        revision = None
-        for line in output.splitlines():
-            if "Last Changed Rev" in line:
-                revision = line.split(":")[1].strip()
-
-        return revision
-
-    def _sortable_revision(self, url, ud, d):
-        """
-        Return a sortable revision number which in our case is the revision number
-        """
-
-        return self._build_revision(url, ud, d)
-
-    def _build_revision(self, url, ud, d):
-        return ud.revision

bitbake/lib/bb/fetch2/wget.py

@@ -1,92 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-"""
-BitBake 'Fetch' implementations
-
-Classes for obtaining upstream sources for the
-BitBake build tools.
-
-"""
-
-# Copyright (C) 2003, 2004  Chris Larson
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-#
-# Based on functions from the base bb module, Copyright 2003 Holger Schurig
-
-import os
-import logging
-import bb
-import urllib
-from   bb import data
-from   bb.fetch2 import FetchMethod
-from   bb.fetch2 import FetchError
-from   bb.fetch2 import encodeurl
-from   bb.fetch2 import decodeurl
-from   bb.fetch2 import logger
-from   bb.fetch2 import runfetchcmd
-
-class Wget(FetchMethod):
-    """Class to fetch urls via 'wget'"""
-    def supports(self, url, ud, d):
-        """
-        Check to see if a given url can be fetched with wget.
-        """
-        return ud.type in ['http', 'https', 'ftp']
-
-    def urldata_init(self, ud, d):
-
-        ud.basename = os.path.basename(ud.path)
-        ud.localfile = data.expand(urllib.unquote(ud.basename), d)
-
-    def download(self, uri, ud, d, checkonly = False):
-        """Fetch urls"""
-
-        def fetch_uri(uri, ud, d):
-            if checkonly:
-                fetchcmd = data.getVar("CHECKCOMMAND", d, True)
-            elif os.path.exists(ud.localpath):
-                # file exists, but we didnt complete it.. trying again..
-                fetchcmd = data.getVar("RESUMECOMMAND", d, True)
-            else:
-                fetchcmd = data.getVar("FETCHCOMMAND", d, True)
-
-            uri = uri.split(";")[0]
-            uri_decoded = list(decodeurl(uri))
-            uri_type = uri_decoded[0]
-            uri_host = uri_decoded[1]
-
-            fetchcmd = fetchcmd.replace("${URI}", uri.split(";")[0])
-            fetchcmd = fetchcmd.replace("${FILE}", ud.basename)
-            if not checkonly:
-                logger.info("fetch " + uri)
-                logger.debug(2, "executing " + fetchcmd)
-            bb.fetch2.check_network_access(d, fetchcmd)
-            runfetchcmd(fetchcmd, d, quiet=checkonly)
-
-            # Sanity check since wget can pretend it succeed when it didn't
-            # Also, this used to happen if sourceforge sent us to the mirror page
-            if not os.path.exists(ud.localpath) and not checkonly:
-                raise FetchError("The fetch command returned success for url %s but %s doesn't exist?!" % (uri, ud.localpath), uri)
-
-        localdata = data.createCopy(d)
-        data.setVar('OVERRIDES', "wget:" + data.getVar('OVERRIDES', localdata), localdata)
-        data.update_data(localdata)
-
-        fetch_uri(uri, ud, localdata)
-        
-        return True
-
-    def checkstatus(self, uri, ud, d):
-        return self.download(uri, ud, d, True)

bitbake/lib/bb/msg.py

@@ -23,121 +23,122 @@ Message handling infrastructure for bitbake
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import sys
-import logging
 import collections
-from itertools import groupby
-import warnings
 import bb
 import bb.event
 
-class BBLogFormatter(logging.Formatter):
-    """Formatter which ensures that our 'plain' messages (logging.INFO + 1) are used as is"""
+debug_level = collections.defaultdict(lambda: 0)
+verbose = False
 
-    DEBUG3 = logging.DEBUG - 2
-    DEBUG2 = logging.DEBUG - 1
-    DEBUG = logging.DEBUG
-    VERBOSE = logging.INFO - 1
-    NOTE = logging.INFO
-    PLAIN = logging.INFO + 1
-    ERROR = logging.ERROR
-    WARNING = logging.WARNING
-    CRITICAL = logging.CRITICAL
+def _NamedTuple(name, fields):
+    Tuple = collections.namedtuple(name, " ".join(fields))
+    return Tuple(*range(len(fields)))
 
-    levelnames = {
-        DEBUG3   : 'DEBUG',
-        DEBUG2   : 'DEBUG',
-        DEBUG   : 'DEBUG',
-        VERBOSE: 'NOTE',
-        NOTE    : 'NOTE',
-        PLAIN  : '',
-        WARNING : 'WARNING',
-        ERROR   : 'ERROR',
-        CRITICAL: 'ERROR',
-    }
-
-    def getLevelName(self, levelno):
-        try:
-            return self.levelnames[levelno]
-        except KeyError:
-            self.levelnames[levelno] = value = 'Level %d' % levelno
-            return value
-
-    def format(self, record):
-        record.levelname = self.getLevelName(record.levelno)
-        if record.levelno == self.PLAIN:
-            msg = record.getMessage()
-        else:
-            msg = logging.Formatter.format(self, record)
-
-        if hasattr(record, 'bb_exc_info'):
-            etype, value, tb = record.bb_exc_info
-            formatted = bb.exceptions.format_exception(etype, value, tb, limit=5)
-            msg += '\n' + ''.join(formatted)
-        return msg
-
-class BBLogFilter(object):
-    def __init__(self, handler, level, debug_domains):
-        self.stdlevel = level
-        self.debug_domains = debug_domains
-        loglevel = level
-        for domain in debug_domains:
-            if debug_domains[domain] < loglevel:
-                loglevel = debug_domains[domain]
-        handler.setLevel(loglevel)
-        handler.addFilter(self)
-
-    def filter(self, record):
-        if record.levelno >= self.stdlevel:
-            return True
-        if record.name in self.debug_domains and record.levelno >= self.debug_domains[record.name]:
-            return True
-        return False
+domain = _NamedTuple("Domain", (
+    "Default",
+    "Build",
+    "Cache",
+    "Collection",
+    "Data",
+    "Depends",
+    "Fetcher",
+    "Parsing",
+    "PersistData",
+    "Provider",
+    "RunQueue",
+    "TaskData",
+    "Util"))
 
 
+class MsgBase(bb.event.Event):
+    """Base class for messages"""
 
+    def __init__(self, msg):
+        self._message = msg
+        bb.event.Event.__init__(self)
+
+class MsgDebug(MsgBase):
+    """Debug Message"""
+
+class MsgNote(MsgBase):
+    """Note Message"""
+
+class MsgWarn(MsgBase):
+    """Warning Message"""
+
+class MsgError(MsgBase):
+    """Error Message"""
+
+class MsgFatal(MsgBase):
+    """Fatal Message"""
+
+class MsgPlain(MsgBase):
+    """General output"""
+
+#
 # Message control functions
 #
 
-loggerDefaultDebugLevel = 0
-loggerDefaultVerbose = False
-loggerDefaultDomains = []
+def set_debug_level(level):
+    for d in domain:
+        debug_level[d] = level
+    debug_level[domain.Default] = level
 
-def init_msgconfig(verbose, debug, debug_domains = []):
-    """
-    Set default verbosity and debug levels config the logger
-    """
-    bb.msg.loggerDefaultDebugLevel = debug
-    bb.msg.loggerDefaultVerbose = verbose
-    bb.msg.loggerDefaultDomains = debug_domains
+def get_debug_level(msgdomain = domain.Default):
+    return debug_level[msgdomain]
 
-def addDefaultlogFilter(handler):
+def set_verbose(level):
+    verbose = level
 
-    debug = loggerDefaultDebugLevel
-    verbose = loggerDefaultVerbose
-    domains = loggerDefaultDomains
-
-    if debug:
-        level = BBLogFormatter.DEBUG - debug + 1
-    elif verbose:
-        level = BBLogFormatter.VERBOSE
-    else:
-        level = BBLogFormatter.NOTE
-
-    debug_domains = {}
-    for (domainarg, iterator) in groupby(domains):
-        dlevel = len(tuple(iterator))
-        debug_domains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
-
-    BBLogFilter(handler, level, debug_domains)
+def set_debug_domains(strdomains):
+    for domainstr in strdomains:
+        for d in domain:
+            if domain._fields[d] == domainstr:
+                debug_level[d] += 1
+                break
+        else:
+            warn(None, "Logging domain %s is not valid, ignoring" % domainstr)
 
 #
 # Message handling functions
 #
 
-def fatal(msgdomain, msg):
-    if msgdomain:
-        logger = logging.getLogger("BitBake.%s" % msgdomain)
-    else:
-        logger = logging.getLogger("BitBake")
-    logger.critical(msg)
+def debug(level, msgdomain, msg, fn = None):
+    if not msgdomain:
+        msgdomain = domain.Default
+
+    if debug_level[msgdomain] >= level:
+        bb.event.fire(MsgDebug(msg), None)
+        if bb.event.useStdout:
+            print('DEBUG: %s' % (msg))
+
+def note(level, msgdomain, msg, fn = None):
+    if not msgdomain:
+        msgdomain = domain.Default
+
+    if level == 1 or verbose or debug_level[msgdomain] >= 1:
+        bb.event.fire(MsgNote(msg), None)
+        if bb.event.useStdout:
+            print('NOTE: %s' % (msg))
+
+def warn(msgdomain, msg, fn = None):
+    bb.event.fire(MsgWarn(msg), None)
+    if bb.event.useStdout:
+        print('WARNING: %s' % (msg))
+
+def error(msgdomain, msg, fn = None):
+    bb.event.fire(MsgError(msg), None)
+    if bb.event.useStdout:
+       print('ERROR: %s' % (msg))
+
+def fatal(msgdomain, msg, fn = None):
+    bb.event.fire(MsgFatal(msg), None)
+
+    if bb.event.useStdout:
+        print('FATAL: %s' % (msg))
     sys.exit(1)
+
+def plain(msg, fn = None):
+    bb.event.fire(MsgPlain(msg), None)
+    if bb.event.useStdout:
+        print(msg)

bitbake/lib/bb/namedtuple_with_abc.py

@@ -1,255 +0,0 @@
-# http://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-class-mix-in-for-named/
-#!/usr/bin/env python
-# Copyright (c) 2011 Jan Kaliszewski (zuo). Available under the MIT License.
-
-"""
-namedtuple_with_abc.py:
-* named tuple mix-in + ABC (abstract base class) recipe,
-* works under Python 2.6, 2.7 as well as 3.x.
-
-Import this module to patch collections.namedtuple() factory function
--- enriching it with the 'abc' attribute (an abstract base class + mix-in
-for named tuples) and decorating it with a wrapper that registers each
-newly created named tuple as a subclass of namedtuple.abc.
-
-How to import:
-    import collections, namedtuple_with_abc
-or:
-    import namedtuple_with_abc
-    from collections import namedtuple
-    # ^ in this variant you must import namedtuple function
-    #   *after* importing namedtuple_with_abc module
-or simply:
-    from namedtuple_with_abc import namedtuple
-
-Simple usage example:
-    class Credentials(namedtuple.abc):
-        _fields = 'username password'
-        def __str__(self):
-            return ('{0.__class__.__name__}'
-                    '(username={0.username}, password=...)'.format(self))
-    print(Credentials("alice", "Alice's password"))
-
-For more advanced examples -- see below the "if __name__ == '__main__':".
-"""
-
-import collections
-from abc import ABCMeta, abstractproperty
-from functools import wraps
-from sys import version_info
-
-__all__ = ('namedtuple',)
-_namedtuple = collections.namedtuple
-
-
-class _NamedTupleABCMeta(ABCMeta):
-    '''The metaclass for the abstract base class + mix-in for named tuples.'''
-    def __new__(mcls, name, bases, namespace):
-        fields = namespace.get('_fields')
-        for base in bases:
-            if fields is not None:
-                break
-            fields = getattr(base, '_fields', None)
-        if not isinstance(fields, abstractproperty):
-            basetuple = _namedtuple(name, fields)
-            bases = (basetuple,) + bases
-            namespace.pop('_fields', None)
-            namespace.setdefault('__doc__', basetuple.__doc__)
-            namespace.setdefault('__slots__', ())
-        return ABCMeta.__new__(mcls, name, bases, namespace)
-
-
-exec(
-    # Python 2.x metaclass declaration syntax
-    """class _NamedTupleABC(object):
-        '''The abstract base class + mix-in for named tuples.'''
-        __metaclass__ = _NamedTupleABCMeta
-        _fields = abstractproperty()""" if version_info[0] < 3 else
-    # Python 3.x metaclass declaration syntax
-    """class _NamedTupleABC(metaclass=_NamedTupleABCMeta):
-        '''The abstract base class + mix-in for named tuples.'''
-        _fields = abstractproperty()"""
-)
-
-
-_namedtuple.abc = _NamedTupleABC
-#_NamedTupleABC.register(type(version_info))  # (and similar, in the future...)
-
-@wraps(_namedtuple)
-def namedtuple(*args, **kwargs):
-    '''Named tuple factory with namedtuple.abc subclass registration.'''
-    cls = _namedtuple(*args, **kwargs)
-    _NamedTupleABC.register(cls)
-    return cls
-
-collections.namedtuple = namedtuple
-
-
-
-
-if __name__ == '__main__':
-
-    '''Examples and explanations'''
-
-    # Simple usage
-
-    class MyRecord(namedtuple.abc):
-        _fields = 'x y z'  # such form will be transformed into ('x', 'y', 'z')
-        def _my_custom_method(self):
-            return list(self._asdict().items())
-    # (the '_fields' attribute belongs to the named tuple public API anyway)
-
-    rec = MyRecord(1, 2, 3)
-    print(rec)
-    print(rec._my_custom_method())
-    print(rec._replace(y=222))
-    print(rec._replace(y=222)._my_custom_method())
-
-    # Custom abstract classes...
-
-    class MyAbstractRecord(namedtuple.abc):
-        def _my_custom_method(self):
-            return list(self._asdict().items())
-
-    try:
-        MyAbstractRecord()  # (abstract classes cannot be instantiated)
-    except TypeError as exc:
-        print(exc)
-
-    class AnotherAbstractRecord(MyAbstractRecord):
-        def __str__(self):
-            return '<<<{0}>>>'.format(super(AnotherAbstractRecord,
-                                            self).__str__())
-
-    # ...and their non-abstract subclasses
-
-    class MyRecord2(MyAbstractRecord):
-        _fields = 'a, b'
-
-    class MyRecord3(AnotherAbstractRecord):
-        _fields = 'p', 'q', 'r'
-
-    rec2 = MyRecord2('foo', 'bar')
-    print(rec2)
-    print(rec2._my_custom_method())
-    print(rec2._replace(b=222))
-    print(rec2._replace(b=222)._my_custom_method())
-
-    rec3 = MyRecord3('foo', 'bar', 'baz')
-    print(rec3)
-    print(rec3._my_custom_method())
-    print(rec3._replace(q=222))
-    print(rec3._replace(q=222)._my_custom_method())
-
-   # You can also subclass non-abstract ones...
-
-    class MyRecord33(MyRecord3):
-        def __str__(self):
-            return '< {0!r}, ..., {0!r} >'.format(self.p, self.r)
-
-    rec33 = MyRecord33('foo', 'bar', 'baz')
-    print(rec33)
-    print(rec33._my_custom_method())
-    print(rec33._replace(q=222))
-    print(rec33._replace(q=222)._my_custom_method())
-
-    # ...and even override the magic '_fields' attribute again
-
-    class MyRecord345(MyRecord3):
-        _fields = 'e f g h i j k'
-
-    rec345 = MyRecord345(1, 2, 3, 4, 3, 2, 1)
-    print(rec345)
-    print(rec345._my_custom_method())
-    print(rec345._replace(f=222))
-    print(rec345._replace(f=222)._my_custom_method())
-
-    # Mixing-in some other classes is also possible:
-
-    class MyMixIn(object):
-        def method(self):
-            return "MyMixIn.method() called"
-        def _my_custom_method(self):
-            return "MyMixIn._my_custom_method() called"
-        def count(self, item):
-            return "MyMixIn.count({0}) called".format(item)
-        def _asdict(self):  # (cannot override a namedtuple method, see below)
-            return "MyMixIn._asdict() called"
-
-    class MyRecord4(MyRecord33, MyMixIn):  # mix-in on the right
-        _fields = 'j k l x'
-
-    class MyRecord5(MyMixIn, MyRecord33):  # mix-in on the left
-        _fields = 'j k l x y'
-
-    rec4 = MyRecord4(1, 2, 3, 2)
-    print(rec4)
-    print(rec4.method())
-    print(rec4._my_custom_method())  # MyRecord33's
-    print(rec4.count(2))  # tuple's
-    print(rec4._replace(k=222))
-    print(rec4._replace(k=222).method())
-    print(rec4._replace(k=222)._my_custom_method())  # MyRecord33's
-    print(rec4._replace(k=222).count(8))  # tuple's
-
-    rec5 = MyRecord5(1, 2, 3, 2, 1)
-    print(rec5)
-    print(rec5.method())
-    print(rec5._my_custom_method())  # MyMixIn's
-    print(rec5.count(2))  # MyMixIn's
-    print(rec5._replace(k=222))
-    print(rec5._replace(k=222).method())
-    print(rec5._replace(k=222)._my_custom_method())  # MyMixIn's
-    print(rec5._replace(k=222).count(2))  # MyMixIn's
-
-    # None that behavior: the standard namedtuple methods cannot be
-    # overriden by a foreign mix-in -- even if the mix-in is declared
-    # as the leftmost base class (but, obviously, you can override them
-    # in the defined class or its subclasses):
-
-    print(rec4._asdict())  # (returns a dict, not "MyMixIn._asdict() called")
-    print(rec5._asdict())  # (returns a dict, not "MyMixIn._asdict() called")
-
-    class MyRecord6(MyRecord33):
-        _fields = 'j k l x y z'
-        def _asdict(self):
-            return "MyRecord6._asdict() called"
-    rec6 = MyRecord6(1, 2, 3, 1, 2, 3)
-    print(rec6._asdict())  # (this returns "MyRecord6._asdict() called")
-
-    # All that record classes are real subclasses of namedtuple.abc:
-
-    assert issubclass(MyRecord, namedtuple.abc)
-    assert issubclass(MyAbstractRecord, namedtuple.abc)
-    assert issubclass(AnotherAbstractRecord, namedtuple.abc)
-    assert issubclass(MyRecord2, namedtuple.abc)
-    assert issubclass(MyRecord3, namedtuple.abc)
-    assert issubclass(MyRecord33, namedtuple.abc)
-    assert issubclass(MyRecord345, namedtuple.abc)
-    assert issubclass(MyRecord4, namedtuple.abc)
-    assert issubclass(MyRecord5, namedtuple.abc)
-    assert issubclass(MyRecord6, namedtuple.abc)
-
-    # ...but abstract ones are not subclasses of tuple
-    # (and this is what you probably want):
-
-    assert not issubclass(MyAbstractRecord, tuple)
-    assert not issubclass(AnotherAbstractRecord, tuple)
-
-    assert issubclass(MyRecord, tuple)
-    assert issubclass(MyRecord2, tuple)
-    assert issubclass(MyRecord3, tuple)
-    assert issubclass(MyRecord33, tuple)
-    assert issubclass(MyRecord345, tuple)
-    assert issubclass(MyRecord4, tuple)
-    assert issubclass(MyRecord5, tuple)
-    assert issubclass(MyRecord6, tuple)
-
-    # Named tuple classes created with namedtuple() factory function
-    # (in the "traditional" way) are registered as "virtual" subclasses
-    # of namedtuple.abc:
-
-    MyTuple = namedtuple('MyTuple', 'a b c')
-    mt = MyTuple(1, 2, 3)
-    assert issubclass(MyTuple, namedtuple.abc)
-    assert isinstance(mt, namedtuple.abc)

bitbake/lib/bb/parse/__init__.py

@@ -26,15 +26,10 @@ File parsers for the BitBake build tools.
 
 handlers = []
 
-import os
-import stat
-import logging
-import bb
+import bb, os
 import bb.utils
 import bb.siggen
 
-logger = logging.getLogger("BitBake.Parsing")
-
 class ParseError(Exception):
     """Exception raised when parsing fails"""
 
@@ -44,19 +39,19 @@ class SkipPackage(Exception):
 __mtime_cache = {}
 def cached_mtime(f):
     if f not in __mtime_cache:
-        __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
+        __mtime_cache[f] = os.stat(f)[8]
     return __mtime_cache[f]
 
 def cached_mtime_noerror(f):
     if f not in __mtime_cache:
         try:
-            __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
+            __mtime_cache[f] = os.stat(f)[8]
         except OSError:
             return 0
     return __mtime_cache[f]
 
 def update_mtime(f):
-    __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
+    __mtime_cache[f] = os.stat(f)[8]
     return __mtime_cache[f]
 
 def mark_dependency(d, f):
@@ -85,18 +80,18 @@ def init(fn, data):
         if h['supports'](fn):
             return h['init'](data)
 
-def init_parser(d):
-    bb.parse.siggen = bb.siggen.init(d)
+def init_parser(d, dumpsigs):
+    bb.parse.siggen = bb.siggen.init(d, dumpsigs)
 
 def resolve_file(fn, d):
     if not os.path.isabs(fn):
         bbpath = bb.data.getVar("BBPATH", d, True)
-        newfn = bb.utils.which(bbpath, fn)
+        newfn = bb.which(bbpath, fn)
         if not newfn:
             raise IOError("file %s not found in %s" % (fn, bbpath))
         fn = newfn
 
-    logger.debug(2, "LOAD %s", fn)
+    bb.msg.debug(2, bb.msg.domain.Parsing, "LOAD %s" % fn)
     return fn
 
 # Used by OpenEmbedded metadata

bitbake/lib/bb/parse/ast.py

@@ -23,14 +23,11 @@
 
 from __future__ import absolute_import
 from future_builtins import filter
-import re
-import string
-import logging
-import bb
-import itertools
+import bb, re, string
 from bb import methodpool
-from bb.parse import logger
+import itertools
 
+__word__ = re.compile(r"\S+")
 __parsed_methods__ = bb.methodpool.get_parsed_dict()
 _bbversions_re = re.compile(r"\[(?P<from>[0-9]+)-(?P<to>[0-9]+)\]")
 
@@ -40,14 +37,13 @@ class StatementGroup(list):
             statement.eval(data)
 
 class AstNode(object):
-    def __init__(self, filename, lineno):
-        self.filename = filename
-        self.lineno = lineno
+    pass
 
 class IncludeNode(AstNode):
-    def __init__(self, filename, lineno, what_file, force):
-        AstNode.__init__(self, filename, lineno)
+    def __init__(self, what_file, fn, lineno, force):
         self.what_file = what_file
+        self.from_fn = fn
+        self.from_lineno = lineno
         self.force = force
 
     def eval(self, data):
@@ -55,17 +51,16 @@ class IncludeNode(AstNode):
         Include the file and evaluate the statements
         """
         s = bb.data.expand(self.what_file, data)
-        logger.debug(2, "CONF %s:%s: including %s", self.filename, self.lineno, s)
+        bb.msg.debug(3, bb.msg.domain.Parsing, "CONF %s:%d: including %s" % (self.from_fn, self.from_lineno, s))
 
         # TODO: Cache those includes... maybe not here though
         if self.force:
-            bb.parse.ConfHandler.include(self.filename, s, data, "include required")
+            bb.parse.ConfHandler.include(self.from_fn, s, data, "include required")
         else:
-            bb.parse.ConfHandler.include(self.filename, s, data, False)
+            bb.parse.ConfHandler.include(self.from_fn, s, data, False)
 
 class ExportNode(AstNode):
-    def __init__(self, filename, lineno, var):
-        AstNode.__init__(self, filename, lineno)
+    def __init__(self, var):
         self.var = var
 
     def eval(self, data):
@@ -78,15 +73,14 @@ class DataNode(AstNode):
     this need to be re-evaluated... we might be able to do
     that faster with multiple classes.
     """
-    def __init__(self, filename, lineno, groupd):
-        AstNode.__init__(self, filename, lineno)
+    def __init__(self, groupd):
         self.groupd = groupd
 
     def getFunc(self, key, data):
         if 'flag' in self.groupd and self.groupd['flag'] != None:
-            return data.getVarFlag(key, self.groupd['flag'], noweakdefault=True)
+            return bb.data.getVarFlag(key, self.groupd['flag'], data)
         else:
-            return data.getVar(key, noweakdefault=True)
+            return bb.data.getVar(key, data)
 
     def eval(self, data):
         groupd = self.groupd
@@ -100,7 +94,7 @@ class DataNode(AstNode):
         elif "colon" in groupd and groupd["colon"] != None:
             e = data.createCopy()
             bb.data.update_data(e)
-            val = bb.data.expand(groupd["value"], e, key + "[:=]")
+            val = bb.data.expand(groupd["value"], e)
         elif "append" in groupd and groupd["append"] != None:
             val = "%s %s" % ((self.getFunc(key, data) or ""), groupd["value"])
         elif "prepend" in groupd and groupd["prepend"] != None:
@@ -115,22 +109,26 @@ class DataNode(AstNode):
         if 'flag' in groupd and groupd['flag'] != None:
             bb.data.setVarFlag(key, groupd['flag'], val, data)
         elif groupd["lazyques"]:
+            assigned = bb.data.getVar("__lazy_assigned", data) or []
+            assigned.append(key)
+            bb.data.setVar("__lazy_assigned", assigned, data)
             bb.data.setVarFlag(key, "defaultval", val, data)
         else:
             bb.data.setVar(key, val, data)
 
-class MethodNode(AstNode):
-    def __init__(self, filename, lineno, func_name, body):
-        AstNode.__init__(self, filename, lineno)
+class MethodNode:
+    def __init__(self, func_name, body, lineno, fn):
         self.func_name = func_name
         self.body = body
+        self.fn = fn
+        self.lineno = lineno
 
     def eval(self, data):
         if self.func_name == "__anonymous":
-            funcname = ("__anon_%s_%s" % (self.lineno, self.filename.translate(string.maketrans('/.+-', '____'))))
+            funcname = ("__anon_%s_%s" % (self.lineno, self.fn.translate(string.maketrans('/.+-', '____'))))
             if not funcname in bb.methodpool._parsed_fns:
                 text = "def %s(d):\n" % (funcname) + '\n'.join(self.body)
-                bb.methodpool.insert_method(funcname, text, self.filename)
+                bb.methodpool.insert_method(funcname, text, self.fn)
             anonfuncs = bb.data.getVar('__BBANONFUNCS', data) or []
             anonfuncs.append(funcname)
             bb.data.setVar('__BBANONFUNCS', anonfuncs, data)
@@ -139,26 +137,25 @@ class MethodNode(AstNode):
             bb.data.setVar(self.func_name, '\n'.join(self.body), data)
 
 class PythonMethodNode(AstNode):
-    def __init__(self, filename, lineno, function, define, body):
-        AstNode.__init__(self, filename, lineno)
-        self.function = function
-        self.define = define
+    def __init__(self, funcname, root, body, fn):
+        self.func_name = funcname
+        self.root = root
         self.body = body
+        self.fn = fn
 
     def eval(self, data):
         # Note we will add root to parsedmethods after having parse
         # 'this' file. This means we will not parse methods from
         # bb classes twice
         text = '\n'.join(self.body)
-        if not bb.methodpool.parsed_module(self.define):
-            bb.methodpool.insert_method(self.define, text, self.filename)
-        bb.data.setVarFlag(self.function, "func", 1, data)
-        bb.data.setVarFlag(self.function, "python", 1, data)
-        bb.data.setVar(self.function, text, data)
+        if not bb.methodpool.parsed_module(self.root):
+            bb.methodpool.insert_method(self.root, text, self.fn)
+        bb.data.setVarFlag(self.func_name, "func", 1, data)
+        bb.data.setVarFlag(self.func_name, "python", 1, data)
+        bb.data.setVar(self.func_name, text, data)
 
 class MethodFlagsNode(AstNode):
-    def __init__(self, filename, lineno, key, m):
-        AstNode.__init__(self, filename, lineno)
+    def __init__(self, key, m):
         self.key = key
         self.m = m
 
@@ -178,9 +175,8 @@ class MethodFlagsNode(AstNode):
             bb.data.delVarFlag(self.key, "fakeroot", data)
 
 class ExportFuncsNode(AstNode):
-    def __init__(self, filename, lineno, fns, classes):
-        AstNode.__init__(self, filename, lineno)
-        self.n = fns.split()
+    def __init__(self, fns, classes):
+        self.n = __word__.findall(fns)
         self.classes = classes
 
     def eval(self, data):
@@ -218,8 +214,7 @@ class ExportFuncsNode(AstNode):
                 bb.data.setVarFlag(var, 'export_func', '1', data)
 
 class AddTaskNode(AstNode):
-    def __init__(self, filename, lineno, func, before, after):
-        AstNode.__init__(self, filename, lineno)
+    def __init__(self, func, before, after):
         self.func = func
         self.before = before
         self.after = after
@@ -250,9 +245,8 @@ class AddTaskNode(AstNode):
                     bb.data.setVarFlag(entry, "deps", [var] + existing, data)
 
 class BBHandlerNode(AstNode):
-    def __init__(self, filename, lineno, fns):
-        AstNode.__init__(self, filename, lineno)
-        self.hs = fns.split()
+    def __init__(self, fns):
+        self.hs = __word__.findall(fns)
 
     def eval(self, data):
         bbhands = bb.data.getVar('__BBHANDLERS', data) or []
@@ -262,58 +256,55 @@ class BBHandlerNode(AstNode):
         bb.data.setVar('__BBHANDLERS', bbhands, data)
 
 class InheritNode(AstNode):
-    def __init__(self, filename, lineno, classes):
-        AstNode.__init__(self, filename, lineno)
-        self.classes = classes
+    def __init__(self, files):
+        self.n = __word__.findall(files)
 
     def eval(self, data):
-        bb.parse.BBHandler.inherit(self.classes, data)
+        bb.parse.BBHandler.inherit(self.n, data)
 
-def handleInclude(statements, filename, lineno, m, force):
-    statements.append(IncludeNode(filename, lineno, m.group(1), force))
+def handleInclude(statements, m, fn, lineno, force):
+    statements.append(IncludeNode(m.group(1), fn, lineno, force))
 
-def handleExport(statements, filename, lineno, m):
-    statements.append(ExportNode(filename, lineno, m.group(1)))
+def handleExport(statements, m):
+    statements.append(ExportNode(m.group(1)))
 
-def handleData(statements, filename, lineno, groupd):
-    statements.append(DataNode(filename, lineno, groupd))
+def handleData(statements, groupd):
+    statements.append(DataNode(groupd))
 
-def handleMethod(statements, filename, lineno, func_name, body):
-    statements.append(MethodNode(filename, lineno, func_name, body))
+def handleMethod(statements, func_name, lineno, fn, body):
+    statements.append(MethodNode(func_name, body, lineno, fn))
 
-def handlePythonMethod(statements, filename, lineno, funcname, root, body):
-    statements.append(PythonMethodNode(filename, lineno, funcname, root, body))
+def handlePythonMethod(statements, funcname, root, body, fn):
+    statements.append(PythonMethodNode(funcname, root, body, fn))
 
-def handleMethodFlags(statements, filename, lineno, key, m):
-    statements.append(MethodFlagsNode(filename, lineno, key, m))
+def handleMethodFlags(statements, key, m):
+    statements.append(MethodFlagsNode(key, m))
 
-def handleExportFuncs(statements, filename, lineno, m, classes):
-    statements.append(ExportFuncsNode(filename, lineno, m.group(1), classes))
+def handleExportFuncs(statements, m, classes):
+    statements.append(ExportFuncsNode(m.group(1), classes))
 
-def handleAddTask(statements, filename, lineno, m):
+def handleAddTask(statements, m):
     func = m.group("func")
     before = m.group("before")
     after = m.group("after")
     if func is None:
         return
 
-    statements.append(AddTaskNode(filename, lineno, func, before, after))
+    statements.append(AddTaskNode(func, before, after))
 
-def handleBBHandlers(statements, filename, lineno, m):
-    statements.append(BBHandlerNode(filename, lineno, m.group(1)))
+def handleBBHandlers(statements, m):
+    statements.append(BBHandlerNode(m.group(1)))
 
-def handleInherit(statements, filename, lineno, m):
-    classes = m.group(1)
-    statements.append(InheritNode(filename, lineno, classes.split()))
+def handleInherit(statements, m):
+    files = m.group(1)
+    n = __word__.findall(files)
+    statements.append(InheritNode(m.group(1)))
 
 def finalize(fn, d, variant = None):
-    all_handlers = {}
-    for var in bb.data.getVar('__BBHANDLERS', d) or []:
-        # try to add the handler
-        handler = bb.data.getVar(var, d)
-        bb.event.register(var, handler)
-
-    bb.event.fire(bb.event.RecipePreFinalise(fn), d)
+    for lazykey in bb.data.getVar("__lazy_assigned", d) or ():
+        if bb.data.getVar(lazykey, d) is None:
+            val = bb.data.getVarFlag(lazykey, "defaultval", d)
+            bb.data.setVar(lazykey, val, d)
 
     bb.data.expandKeys(d)
     bb.data.update_data(d)
@@ -323,6 +314,12 @@ def finalize(fn, d, variant = None):
     bb.utils.simple_exec("\n".join(code), {"d": d})
     bb.data.update_data(d)
 
+    all_handlers = {}
+    for var in bb.data.getVar('__BBHANDLERS', d) or []:
+        # try to add the handler
+        handler = bb.data.getVar(var, d)
+        bb.event.register(var, handler)
+
     tasklist = bb.data.getVar('__BBTASKS', d) or []
     bb.build.add_tasks(tasklist, d)
 
@@ -368,17 +365,15 @@ def _expand_versions(versions):
 def multi_finalize(fn, d):
     appends = (d.getVar("__BBAPPEND", True) or "").split()
     for append in appends:
-        logger.debug(2, "Appending .bbappend file %s to %s", append, fn)
+        bb.msg.debug(2, bb.msg.domain.Parsing, "Appending .bbappend file " + append + " to " + fn)
         bb.parse.BBHandler.handle(append, d, True)
 
-    onlyfinalise = d.getVar("__ONLYFINALISE", False)
-
     safe_d = d
     d = bb.data.createCopy(safe_d)
     try:
         finalize(fn, d)
-    except bb.parse.SkipPackage as e:
-        bb.data.setVar("__SKIPPED", e.args[0], d)
+    except bb.parse.SkipPackage:
+        bb.data.setVar("__SKIPPED", True, d)
     datastores = {"": safe_d}
 
     versions = (d.getVar("BBVERSIONS", True) or "").split()
@@ -420,48 +415,27 @@ def multi_finalize(fn, d):
             verfunc(pv, d, safe_d)
             try:
                 finalize(fn, d)
-            except bb.parse.SkipPackage as e:
-                bb.data.setVar("__SKIPPED", e.args[0], d)
+            except bb.parse.SkipPackage:
+                bb.data.setVar("__SKIPPED", True, d)
 
         _create_variants(datastores, versions, verfunc)
 
     extended = d.getVar("BBCLASSEXTEND", True) or ""
     if extended:
-        # the following is to support bbextends with arguments, for e.g. multilib
-        # an example is as follows:
-        #   BBCLASSEXTEND = "multilib:lib32"
-        # it will create foo-lib32, inheriting multilib.bbclass and set
-        # BBEXTENDCURR to "multilib" and BBEXTENDVARIANT to "lib32"
-        extendedmap = {}
-        variantmap = {}
-
-        for ext in extended.split():
-            eext = ext.split(':', 2)
-            if len(eext) > 1:
-                extendedmap[ext] = eext[0]
-                variantmap[ext] = eext[1]
-            else:
-                extendedmap[ext] = ext
-
         pn = d.getVar("PN", True)
         def extendfunc(name, d):
-            if name != extendedmap[name]:
-                d.setVar("BBEXTENDCURR", extendedmap[name])
-                d.setVar("BBEXTENDVARIANT", variantmap[name])
-            else:
-                d.setVar("PN", "%s-%s" % (pn, name))
-            bb.parse.BBHandler.inherit([extendedmap[name]], d)
+            d.setVar("PN", "%s-%s" % (pn, name))
+            bb.parse.BBHandler.inherit([name], d)
 
         safe_d.setVar("BBCLASSEXTEND", extended)
-        _create_variants(datastores, extendedmap.keys(), extendfunc)
+        _create_variants(datastores, extended.split(), extendfunc)
 
     for variant, variant_d in datastores.iteritems():
         if variant:
             try:
-                if not onlyfinalise or variant in onlyfinalise:
-                    finalize(fn, variant_d, variant)
-            except bb.parse.SkipPackage as e:
-                bb.data.setVar("__SKIPPED", e.args[0], variant_d)
+                finalize(fn, variant_d, variant)
+            except bb.parse.SkipPackage:
+                bb.data.setVar("__SKIPPED", True, variant_d)
 
     if len(datastores) > 1:
         variants = filter(None, datastores.iterkeys())

bitbake/lib/bb/parse/parse_py/BBHandler.py

@@ -27,12 +27,11 @@
 
 from __future__ import absolute_import
 import re, bb, os
-import logging
-import bb.build, bb.utils
+import bb.fetch, bb.build, bb.utils
 from bb import data
 
 from . import ConfHandler
-from .. import resolve_file, ast, logger
+from .. import resolve_file, ast
 from .ConfHandler import include, init
 
 # For compatibility
@@ -65,8 +64,7 @@ IN_PYTHON_EOF = -9999999999999
 
 
 def supports(fn, d):
-    """Return True if fn has a supported extension"""
-    return os.path.splitext(fn)[-1] in [".bb", ".bbclass", ".inc"]
+    return fn[-3:] == ".bb" or fn[-8:] == ".bbclass" or fn[-4:] == ".inc"
 
 def inherit(files, d):
     __inherit_cache = data.getVar('__inherit_cache', d) or []
@@ -74,11 +72,11 @@ def inherit(files, d):
     lineno = 0
     for file in files:
         file = data.expand(file, d)
-        if not os.path.isabs(file) and not file.endswith(".bbclass"):
+        if file[0] != "/" and file[-8:] != ".bbclass":
             file = os.path.join('classes', '%s.bbclass' % file)
 
         if not file in __inherit_cache:
-            logger.log(logging.DEBUG -1, "BB %s:%d: inheriting %s", fn, lineno, file)
+            bb.msg.debug(2, bb.msg.domain.Parsing, "BB %s:%d: inheriting %s" % (fn, lineno, file))
             __inherit_cache.append( file )
             data.setVar('__inherit_cache', __inherit_cache, d)
             include(fn, file, d, "inherit")
@@ -117,12 +115,12 @@ def handle(fn, d, include):
 
 
     if include == 0:
-        logger.debug(2, "BB %s: handle(data)", fn)
+        bb.msg.debug(2, bb.msg.domain.Parsing, "BB " + fn + ": handle(data)")
     else:
-        logger.debug(2, "BB %s: handle(data, include)", fn)
+        bb.msg.debug(2, bb.msg.domain.Parsing, "BB " + fn + ": handle(data, include)")
 
-    base_name = os.path.basename(fn)
-    (root, ext) = os.path.splitext(base_name)
+    (root, ext) = os.path.splitext(os.path.basename(fn))
+    base_name = "%s%s" % (root, ext)
     init(d)
 
     if ext == ".bbclass":
@@ -148,7 +146,7 @@ def handle(fn, d, include):
 
     # DONE WITH PARSING... time to evaluate
     if ext != ".bbclass":
-        data.setVar('FILE', abs_fn, d)
+        data.setVar('FILE', fn, d)
 
     statements.eval(d)
 
@@ -172,7 +170,7 @@ def feeder(lineno, s, fn, root, statements):
     if __infunc__:
         if s == '}':
             __body__.append('')
-            ast.handleMethod(statements, fn, lineno, __infunc__, __body__)
+            ast.handleMethod(statements, __infunc__, lineno, fn, __body__)
             __infunc__ = ""
             __body__ = []
         else:
@@ -185,38 +183,28 @@ def feeder(lineno, s, fn, root, statements):
             __body__.append(s)
             return
         else:
-            ast.handlePythonMethod(statements, fn, lineno, __inpython__,
-                                   root, __body__)
+            ast.handlePythonMethod(statements, __inpython__, root, __body__, fn)
             __body__ = []
             __inpython__ = False
 
             if lineno == IN_PYTHON_EOF:
                 return
 
+#           fall through
 
-    if s and s[0] == '#':
-        if len(__residue__) != 0 and __residue__[0][0] != "#":
-            bb.error("There is a comment on line %s of file %s (%s) which is in the middle of a multiline expression.\nBitbake used to ignore these but no longer does so, please fix your metadata as errors are likely as a result of this change." % (lineno, fn, s))
+    if s == '' or s[0] == '#': return          # skip comments and empty lines
 
-    if s and s[-1] == '\\':
+    if s[-1] == '\\':
         __residue__.append(s[:-1])
         return
 
     s = "".join(__residue__) + s
     __residue__ = []
 
-    # Skip empty lines
-    if s == '':
-        return   
-
-    # Skip comments
-    if s[0] == '#':
-        return
-
     m = __func_start_regexp__.match(s)
     if m:
         __infunc__ = m.group("func") or "__anonymous"
-        ast.handleMethodFlags(statements, fn, lineno, __infunc__, m)
+        ast.handleMethodFlags(statements, __infunc__, m)
         return
 
     m = __def_regexp__.match(s)
@@ -228,22 +216,22 @@ def feeder(lineno, s, fn, root, statements):
 
     m = __export_func_regexp__.match(s)
     if m:
-        ast.handleExportFuncs(statements, fn, lineno, m, classes)
+        ast.handleExportFuncs(statements, m, classes)
         return
 
     m = __addtask_regexp__.match(s)
     if m:
-        ast.handleAddTask(statements, fn, lineno, m)
+        ast.handleAddTask(statements, m)
         return
 
     m = __addhandler_regexp__.match(s)
     if m:
-        ast.handleBBHandlers(statements, fn, lineno, m)
+        ast.handleBBHandlers(statements, m)
         return
 
     m = __inherit_regexp__.match(s)
     if m:
-        ast.handleInherit(statements, fn, lineno, m)
+        ast.handleInherit(statements, m)
         return
 
     return ConfHandler.feeder(lineno, s, fn, statements)

bitbake/lib/bb/parse/parse_py/ConfHandler.py

@@ -25,9 +25,8 @@
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import re, bb.data, os
-import logging
 import bb.utils
-from bb.parse import ParseError, resolve_file, ast, logger
+from bb.parse import ParseError, resolve_file, ast
 
 #__config_regexp__  = re.compile( r"(?P<exp>export\s*)?(?P<var>[a-zA-Z0-9\-_+.${}]+)\s*(?P<colon>:)?(?P<ques>\?)?=\s*(?P<apo>['\"]?)(?P<value>.*)(?P=apo)$")
 __config_regexp__  = re.compile( r"(?P<exp>export\s*)?(?P<var>[a-zA-Z0-9\-_+.${}/]+)(\[(?P<flag>[a-zA-Z0-9\-_+.]+)\])?\s*((?P<colon>:=)|(?P<lazyques>\?\?=)|(?P<ques>\?=)|(?P<append>\+=)|(?P<prepend>=\+)|(?P<predot>=\.)|(?P<postdot>\.=)|=)\s*(?P<apo>['\"]?)(?P<value>.*)(?P=apo)$")
@@ -46,10 +45,10 @@ def supports(fn, d):
 
 def include(oldfn, fn, data, error_out):
     """
-    error_out If True a ParseError will be raised if the to be included
-    config-files could not be included.
+
+    error_out If True a ParseError will be reaised if the to be included
     """
-    if oldfn == fn: # prevent infinite recursion
+    if oldfn == fn: # prevent infinate recursion
         return None
 
     import bb
@@ -69,7 +68,7 @@ def include(oldfn, fn, data, error_out):
     except IOError:
         if error_out:
             raise ParseError("Could not %(error_out)s file %(fn)s" % vars() )
-        logger.debug(2, "CONF file '%s' not found", fn)
+        bb.msg.debug(2, bb.msg.domain.Parsing, "CONF file '%s' not found" % fn)
 
 def handle(fn, data, include):
     init(data)
@@ -96,13 +95,13 @@ def handle(fn, data, include):
         s = s.rstrip()
         if s[0] == '#': continue    # skip comments
         while s[-1] == '\\':
-            s2 = f.readline().strip()
+            s2 = f.readline()[:-1].strip()
             lineno = lineno + 1
             s = s[:-1] + s2
         feeder(lineno, s, fn, statements)
 
     # DONE WITH PARSING... time to evaluate
-    bb.data.setVar('FILE', abs_fn, data)
+    bb.data.setVar('FILE', fn, data)
     statements.eval(data)
     if oldfile:
         bb.data.setVar('FILE', oldfile, data)
@@ -113,22 +112,22 @@ def feeder(lineno, s, fn, statements):
     m = __config_regexp__.match(s)
     if m:
         groupd = m.groupdict()
-        ast.handleData(statements, fn, lineno, groupd)
+        ast.handleData(statements, groupd)
         return
 
     m = __include_regexp__.match(s)
     if m:
-        ast.handleInclude(statements, fn, lineno, m, False)
+        ast.handleInclude(statements, m, fn, lineno, False)
         return
 
     m = __require_regexp__.match(s)
     if m:
-        ast.handleInclude(statements, fn, lineno, m, True)
+        ast.handleInclude(statements, m, fn, lineno, True)
         return
 
     m = __export_regexp__.match(s)
     if m:
-        ast.handleExport(statements, fn, lineno, m)
+        ast.handleExport(statements, m)
         return
 
     raise ParseError("%s:%d: unparsed line: '%s'" % (fn, lineno, s));

bitbake/lib/bb/persist_data.py

@@ -1,12 +1,6 @@
-"""BitBake Persistent Data Store
-
-Used to store data in a central location such that other threads/tasks can
-access them at some future date.  Acts as a convenience wrapper around sqlite,
-currently, providing a key/value store accessed by 'domain'.
-"""
-
+# BitBake Persistent Data Store
+#
 # Copyright (C) 2007        Richard Purdie
-# Copyright (C) 2010        Chris Larson <chris_larson@mentor.com>
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 2 as
@@ -21,187 +15,119 @@ currently, providing a key/value store accessed by 'domain'.
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
-import collections
-import logging
-import os.path
-import sys
-import warnings
-from bb.compat import total_ordering
-from collections import Mapping
+import bb, os
+import bb.utils
 
 try:
     import sqlite3
 except ImportError:
-    from pysqlite2 import dbapi2 as sqlite3
+    try:
+        from pysqlite2 import dbapi2 as sqlite3
+    except ImportError:
+        bb.msg.fatal(bb.msg.domain.PersistData, "Importing sqlite3 and pysqlite2 failed, please install one of them. Python 2.5 or a 'python-pysqlite2' like package is likely to be what you need.")
 
 sqlversion = sqlite3.sqlite_version_info
 if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
-    raise Exception("sqlite3 version 3.3.0 or later is required.")
+    bb.msg.fatal(bb.msg.domain.PersistData, "sqlite3 version 3.3.0 or later is required.")
 
+class PersistData:
+    """
+    BitBake Persistent Data Store
 
-logger = logging.getLogger("BitBake.PersistData")
-if hasattr(sqlite3, 'enable_shared_cache'):
-    sqlite3.enable_shared_cache(True)
+    Used to store data in a central location such that other threads/tasks can
+    access them at some future date.
 
+    The "domain" is used as a key to isolate each data pool and in this
+    implementation corresponds to an SQL table. The SQL table consists of a
+    simple key and value pair.
 
-@total_ordering
-class SQLTable(collections.MutableMapping):
-    """Object representing a table/domain in the database"""
-    def __init__(self, cachefile, table):
-        self.cachefile = cachefile
-        self.table = table
-        self.cursor = connect(self.cachefile)
+    Why sqlite? It handles all the locking issues for us.
+    """
+    def __init__(self, d, persistent_database_connection):
+        if "connection" in persistent_database_connection:
+            self.cursor = persistent_database_connection["connection"].cursor()
+            return
+        self.cachedir = bb.data.getVar("PERSISTENT_DIR", d, True) or bb.data.getVar("CACHE", d, True)
+        if self.cachedir in [None, '']:
+            bb.msg.fatal(bb.msg.domain.PersistData, "Please set the 'PERSISTENT_DIR' or 'CACHE' variable.")
+        try:
+            os.stat(self.cachedir)
+        except OSError:
+            bb.utils.mkdirhier(self.cachedir)
 
-        self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value TEXT);"
-                      % table)
+        self.cachefile = os.path.join(self.cachedir, "bb_persist_data.sqlite3")
+        bb.msg.debug(1, bb.msg.domain.PersistData, "Using '%s' as the persistent data cache" % self.cachefile)
 
-    def _execute(self, *query):
-        """Execute a query, waiting to acquire a lock if necessary"""
-        count = 0
-        while True:
-            try:
-                return self.cursor.execute(*query)
-            except sqlite3.OperationalError as exc:
-                if 'database is locked' in str(exc) and count < 500:
-                    count = count + 1
-                    self.cursor.close()
-                    self.cursor = connect(self.cachefile)
-                    continue
-                raise
-
-    def __enter__(self):
-        self.cursor.__enter__()
-        return self
-
-    def __exit__(self, *excinfo):
-        self.cursor.__exit__(*excinfo)
-
-    def __getitem__(self, key):
-        data = self._execute("SELECT * from %s where key=?;" %
-                             self.table, [key])
-        for row in data:
-            return row[1]
-        raise KeyError(key)
-
-    def __delitem__(self, key):
-        if key not in self:
-            raise KeyError(key)
-        self._execute("DELETE from %s where key=?;" % self.table, [key])
-
-    def __setitem__(self, key, value):
-        if not isinstance(key, basestring):
-            raise TypeError('Only string keys are supported')
-        elif not isinstance(value, basestring):
-            raise TypeError('Only string values are supported')
-
-        data = self._execute("SELECT * from %s where key=?;" %
-                                   self.table, [key])
-        exists = len(list(data))
-        if exists:
-            self._execute("UPDATE %s SET value=? WHERE key=?;" % self.table,
-                          [value, key])
-        else:
-            self._execute("INSERT into %s(key, value) values (?, ?);" %
-                          self.table, [key, value])
-
-    def __contains__(self, key):
-        return key in set(self)
-
-    def __len__(self):
-        data = self._execute("SELECT COUNT(key) FROM %s;" % self.table)
-        for row in data:
-            return row[0]
-
-    def __iter__(self):
-        data = self._execute("SELECT key FROM %s;" % self.table)
-        return (row[0] for row in data)
-
-    def __lt__(self, other):
-        if not isinstance(other, Mapping):
-            raise NotImplemented
-
-        return len(self) < len(other)
-
-    def values(self):
-        return list(self.itervalues())
-
-    def itervalues(self):
-        data = self._execute("SELECT value FROM %s;" % self.table)
-        return (row[0] for row in data)
-
-    def items(self):
-        return list(self.iteritems())
-
-    def iteritems(self):
-        return self._execute("SELECT * FROM %s;" % self.table)
-
-    def clear(self):
-        self._execute("DELETE FROM %s;" % self.table)
-
-    def has_key(self, key):
-        return key in self
-
-
-class PersistData(object):
-    """Deprecated representation of the bitbake persistent data store"""
-    def __init__(self, d):
-        warnings.warn("Use of PersistData is deprecated.  Please use "
-                      "persist(domain, d) instead.",
-                      category=DeprecationWarning,
-                      stacklevel=2)
-
-        self.data = persist(d)
-        logger.debug(1, "Using '%s' as the persistent data cache",
-                     self.data.filename)
+        connection = sqlite3.connect(self.cachefile, timeout=5, isolation_level=None)
+        persistent_database_connection["connection"] = connection
+        self.cursor = persistent_database_connection["connection"].cursor()
 
     def addDomain(self, domain):
         """
-        Add a domain (pending deprecation)
+        Should be called before any domain is used
+        Creates it if it doesn't exist.
         """
-        return self.data[domain]
+        self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value TEXT);" % domain)
 
     def delDomain(self, domain):
         """
         Removes a domain and all the data it contains
         """
-        del self.data[domain]
+        self._execute("DROP TABLE IF EXISTS %s;" % domain)
 
     def getKeyValues(self, domain):
         """
         Return a list of key + value pairs for a domain
         """
-        return self.data[domain].items()
+        ret = {}
+        data = self._execute("SELECT key, value from %s;" % domain)
+        for row in data:
+            ret[str(row[0])] = str(row[1])
+
+        return ret
 
     def getValue(self, domain, key):
         """
         Return the value of a key for a domain
         """
-        return self.data[domain][key]
+        data = self._execute("SELECT * from %s where key=?;" % domain, [key])
+        for row in data:
+            return row[1]
 
     def setValue(self, domain, key, value):
         """
         Sets the value of a key for a domain
         """
-        self.data[domain][key] = value
+        data = self._execute("SELECT * from %s where key=?;" % domain, [key])
+        rows = 0
+        for row in data:
+            rows = rows + 1
+        if rows:
+            self._execute("UPDATE %s SET value=? WHERE key=?;" % domain, [value, key])
+        else:
+            self._execute("INSERT into %s(key, value) values (?, ?);" % domain, [key, value])
 
     def delValue(self, domain, key):
         """
         Deletes a key/value pair
         """
-        del self.data[domain][key]
+        self._execute("DELETE from %s where key=?;" % domain, [key])
 
-def connect(database):
-    return sqlite3.connect(database, timeout=5, isolation_level=None)
-
-def persist(domain, d):
-    """Convenience factory for SQLTable objects based upon metadata"""
-    import bb.data, bb.utils
-    cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or
-                bb.data.getVar("CACHE", d, True))
-    if not cachedir:
-        logger.critical("Please set the 'PERSISTENT_DIR' or 'CACHE' variable")
-        sys.exit(1)
-
-    bb.utils.mkdirhier(cachedir)
-    cachefile = os.path.join(cachedir, "bb_persist_data.sqlite3")
-    return SQLTable(cachefile, domain)
+    #
+    # We wrap the sqlite execute calls as on contended machines or single threaded 
+    # systems we can have multiple processes trying to access the DB at once and it seems
+    # sqlite sometimes doesn't wait for the timeout. We therefore loop but put in an 
+    # emergency brake too
+    #
+    def _execute(self, *query):
+        count = 0
+        while True:
+            try:
+                ret = self.cursor.execute(*query)
+                #print "Had to retry %s times" % count
+                return ret
+            except sqlite3.OperationalError as e:
+                if 'database is locked' in str(e) and count < 500:
+                    count = count + 1
+                    continue
+                raise

bitbake/lib/bb/process.py

@@ -1,109 +0,0 @@
-import logging
-import signal
-import subprocess
-
-logger = logging.getLogger('BitBake.Process')
-
-def subprocess_setup():
-    # Python installs a SIGPIPE handler by default. This is usually not what
-    # non-Python subprocesses expect.
-    signal.signal(signal.SIGPIPE, signal.SIG_DFL)
-
-class CmdError(RuntimeError):
-    def __init__(self, command, msg=None):
-        self.command = command
-        self.msg = msg
-
-    def __str__(self):
-        if not isinstance(self.command, basestring):
-            cmd = subprocess.list2cmdline(self.command)
-        else:
-            cmd = self.command
-
-        msg = "Execution of '%s' failed" % cmd
-        if self.msg:
-            msg += ': %s' % self.msg
-        return msg
-
-class NotFoundError(CmdError):
-    def __str__(self):
-        return CmdError.__str__(self) + ": command not found"
-
-class ExecutionError(CmdError):
-    def __init__(self, command, exitcode, stdout = None, stderr = None):
-        CmdError.__init__(self, command)
-        self.exitcode = exitcode
-        self.stdout = stdout
-        self.stderr = stderr
-
-    def __str__(self):
-        message = ""
-        if self.stderr:
-            message += self.stderr
-        if self.stdout:
-            message += self.stdout
-        if message:
-            message = ":\n" + message
-        return (CmdError.__str__(self) +
-                " with exit code %s" % self.exitcode + message)
-
-class Popen(subprocess.Popen):
-    defaults = {
-        "close_fds": True,
-        "preexec_fn": subprocess_setup,
-        "stdout": subprocess.PIPE,
-        "stderr": subprocess.STDOUT,
-        "stdin": subprocess.PIPE,
-        "shell": False,
-    }
-
-    def __init__(self, *args, **kwargs):
-        options = dict(self.defaults)
-        options.update(kwargs)
-        subprocess.Popen.__init__(self, *args, **options)
-
-def _logged_communicate(pipe, log, input):
-    if pipe.stdin:
-        if input is not None:
-            pipe.stdin.write(input)
-        pipe.stdin.close()
-
-    bufsize = 512
-    outdata, errdata = [], []
-    while pipe.poll() is None:
-        if pipe.stdout is not None:
-            data = pipe.stdout.read(bufsize)
-            if data is not None:
-                outdata.append(data)
-                log.write(data)
-
-        if pipe.stderr is not None:
-            data = pipe.stderr.read(bufsize)
-            if data is not None:
-                errdata.append(data)
-                log.write(data)
-    return ''.join(outdata), ''.join(errdata)
-
-def run(cmd, input=None, log=None, **options):
-    """Convenience function to run a command and return its output, raising an
-    exception when the command fails"""
-
-    if isinstance(cmd, basestring) and not "shell" in options:
-        options["shell"] = True
-
-    try:
-        pipe = Popen(cmd, **options)
-    except OSError as exc:
-        if exc.errno == 2:
-            raise NotFoundError(cmd)
-        else:
-            raise CmdError(cmd, exc)
-
-    if log:
-        stdout, stderr = _logged_communicate(pipe, log, input)
-    else:
-        stdout, stderr = pipe.communicate(input)
-
-    if pipe.returncode != 0:
-        raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
-    return stdout, stderr

bitbake/lib/bb/providers.py

@@ -22,12 +22,9 @@
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import re
-import logging
 from bb import data, utils
 import bb
 
-logger = logging.getLogger("BitBake.Provider")
-
 class NoProvider(Exception):
     """Exception raised when no provider of a build dependency can be found"""
 
@@ -84,10 +81,10 @@ def findPreferredProvider(pn, cfgData, dataCache, pkg_pn = None, item = None):
     preferred_ver = None
 
     localdata = data.createCopy(cfgData)
-    bb.data.setVar('OVERRIDES', "%s:pn-%s:%s" % (data.getVar('OVERRIDES', localdata), pn, pn), localdata)
+    bb.data.setVar('OVERRIDES', "pn-%s:%s:%s" % (pn, pn, data.getVar('OVERRIDES', localdata)), localdata)
     bb.data.update_data(localdata)
 
-    preferred_v = bb.data.getVar('PREFERRED_VERSION', localdata, True)
+    preferred_v = bb.data.getVar('PREFERRED_VERSION_%s' % pn, localdata, True)
     if preferred_v:
         m = re.match('(\d+:)*(.*)(_.*)*', preferred_v)
         if m:
@@ -123,21 +120,9 @@ def findPreferredProvider(pn, cfgData, dataCache, pkg_pn = None, item = None):
         if item:
             itemstr = " (for item %s)" % item
         if preferred_file is None:
-            logger.info("preferred version %s of %s not available%s", pv_str, pn, itemstr)
-            available_vers = []
-            for file_set in pkg_pn:
-                for f in file_set:
-                    pe, pv, pr = dataCache.pkg_pepvpr[f]
-                    ver_str = pv
-                    if pe:
-                        ver_str = "%s:%s" % (pe, ver_str)
-                    if not ver_str in available_vers:
-                        available_vers.append(ver_str)
-            if available_vers:
-                available_vers.sort()
-                logger.info("versions of %s available: %s", pn, ' '.join(available_vers))
+            bb.msg.note(1, bb.msg.domain.Provider, "preferred version %s of %s not available%s" % (pv_str, pn, itemstr))
         else:
-            logger.debug(1, "selecting %s as PREFERRED_VERSION %s of package %s%s", preferred_file, pv_str, pn, itemstr)
+            bb.msg.debug(1, bb.msg.domain.Provider, "selecting %s as PREFERRED_VERSION %s of package %s%s" % (preferred_file, pv_str, pn, itemstr))
 
     return (preferred_ver, preferred_file)
 
@@ -204,7 +189,7 @@ def _filterProviders(providers, item, cfgData, dataCache):
             pkg_pn[pn] = []
         pkg_pn[pn].append(p)
 
-    logger.debug(1, "providers for %s are: %s", item, pkg_pn.keys())
+    bb.msg.debug(1, bb.msg.domain.Provider, "providers for %s are: %s" % (item, pkg_pn.keys()))
 
     # First add PREFERRED_VERSIONS
     for pn in pkg_pn:
@@ -221,7 +206,7 @@ def _filterProviders(providers, item, cfgData, dataCache):
         eligible.append(preferred_versions[pn][1])
 
     if len(eligible) == 0:
-        logger.error("no eligible providers for %s", item)
+        bb.msg.error(bb.msg.domain.Provider, "no eligible providers for %s" % item)
         return 0
 
     # If pn == item, give it a slight default preference
@@ -257,13 +242,13 @@ def filterProviders(providers, item, cfgData, dataCache):
         for p in eligible:
             pn = dataCache.pkg_fn[p]
             if dataCache.preferred[item] == pn:
-                logger.verbose("selecting %s to satisfy %s due to PREFERRED_PROVIDERS", pn, item)
+                bb.msg.note(2, bb.msg.domain.Provider, "selecting %s to satisfy %s due to PREFERRED_PROVIDERS" % (pn, item))
                 eligible.remove(p)
                 eligible = [p] + eligible
                 foundUnique = True
                 break
 
-    logger.debug(1, "sorted providers for %s are: %s", item, eligible)
+    bb.msg.debug(1, bb.msg.domain.Provider, "sorted providers for %s are: %s" % (item, eligible))
 
     return eligible, foundUnique
 
@@ -279,31 +264,27 @@ def filterProvidersRunTime(providers, item, cfgData, dataCache):
     # Should use dataCache.preferred here?
     preferred = []
     preferred_vars = []
-    pns = {}
-    for p in eligible:
-        pns[dataCache.pkg_fn[p]] = p
     for p in eligible:
         pn = dataCache.pkg_fn[p]
         provides = dataCache.pn_provides[pn]
         for provide in provides:
+            bb.msg.note(2, bb.msg.domain.Provider, "checking PREFERRED_PROVIDER_%s" % (provide))
             prefervar = bb.data.getVar('PREFERRED_PROVIDER_%s' % provide, cfgData, 1)
-            logger.debug(1, "checking PREFERRED_PROVIDER_%s (value %s) against %s", provide, prefervar, pns.keys())
-            if prefervar in pns and pns[prefervar] not in preferred:
+            if prefervar == pn:
                 var = "PREFERRED_PROVIDER_%s = %s" % (provide, prefervar)
-                logger.verbose("selecting %s to satisfy runtime %s due to %s", prefervar, item, var)
+                bb.msg.note(2, bb.msg.domain.Provider, "selecting %s to satisfy runtime %s due to %s" % (pn, item, var))
                 preferred_vars.append(var)
-                pref = pns[prefervar]
-                eligible.remove(pref)
-                eligible = [pref] + eligible
-                preferred.append(pref)
+                eligible.remove(p)
+                eligible = [p] + eligible
+                preferred.append(p)
                 break
 
     numberPreferred = len(preferred)
 
     if numberPreferred > 1:
-        logger.error("Trying to resolve runtime dependency %s resulted in conflicting PREFERRED_PROVIDER entries being found.\nThe providers found were: %s\nThe PREFERRED_PROVIDER entries resulting in this conflict were: %s", item, preferred, preferred_vars)
+        bb.msg.error(bb.msg.domain.Provider, "Conflicting PREFERRED_PROVIDER entries were found which resulted in an attempt to select multiple providers (%s) for runtime dependecy %s\nThe entries resulting in this conflict were: %s" % (preferred, item, preferred_vars))
 
-    logger.debug(1, "sorted providers for %s are: %s", item, eligible)
+    bb.msg.debug(1, bb.msg.domain.Provider, "sorted providers for %s are: %s" % (item, eligible))
 
     return eligible, numberPreferred
 
@@ -333,7 +314,7 @@ def getRuntimeProviders(dataCache, rdepend):
             try:
                 regexp = re.compile(pattern)
             except:
-                logger.error("Error parsing regular expression '%s'", pattern)
+                bb.msg.error(bb.msg.domain.Provider, "Error parsing re expression: %s" % pattern)
                 raise
             regexp_cache[pattern] = regexp
         if regexp.match(rdepend):

bitbake/lib/bb/server/none.py

@@ -18,24 +18,33 @@
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 """
-    This module implements a passthrough server for BitBake.
+    This module implements an xmlrpc server for BitBake.
 
-    Use register_idle_function() to add a function which the server
-    calls from within idle_commands when no requests are pending. Make sure
+    Use this by deriving a class from BitBakeXMLRPCServer and then adding
+    methods which you want to "export" via XMLRPC. If the methods have the
+    prefix xmlrpc_, then registering those function will happen automatically,
+    if not, you need to call register_function.
+
+    Use register_idle_function() to add a function which the xmlrpc server
+    calls from within server_forever when no requests are pending. Make sure
     that those functions are non-blocking or else you will introduce latency
     in the server's main loop.
 """
 
 import time
 import bb
-import signal
+from bb.ui import uievent
+import xmlrpclib
+import pickle
 
 DEBUG = False
 
+from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
 import inspect, select
 
 class BitBakeServerCommands():
-    def __init__(self, server):
+    def __init__(self, server, cooker):
+        self.cooker = cooker
         self.server = server
 
     def runCommand(self, command):
@@ -67,7 +76,7 @@ class BBUIEventQueue:
             self.parent = parent
         @staticmethod
         def send(event):
-            bb.server.none.eventQueue.append(event)
+            bb.server.none.eventQueue.append(pickle.loads(event))
         @staticmethod
         def quit():
             return
@@ -77,22 +86,18 @@ class BBUIEventQueue:
         self.BBServer = BBServer
         self.EventHandle = bb.event.register_UIHhandler(self)
 
-    def __popEvent(self):
-        if len(self.eventQueue) == 0:
-            return None
-        return self.eventQueue.pop(0)
-
     def getEvent(self):
         if len(self.eventQueue) == 0:
-          self.BBServer.idle_commands(0)
-        return self.__popEvent()
+            return None
+
+        return self.eventQueue.pop(0)
 
     def waitEvent(self, delay):
-        event = self.__popEvent()
+        event = self.getEvent()
         if event:
             return event
         self.BBServer.idle_commands(delay)
-        return self.__popEvent()
+        return self.getEvent()
 
     def queue_event(self, event):
         self.eventQueue.append(event)
@@ -100,21 +105,13 @@ class BBUIEventQueue:
     def system_quit( self ):
         bb.event.unregister_UIHhandler(self.EventHandle)
 
-# Dummy signal handler to ensure we break out of sleep upon SIGCHLD
-def chldhandler(signum, stackframe):
-    pass
-
-class BitBakeNoneServer():
+class BitBakeServer():
     # remove this when you're done with debugging
     # allow_reuse_address = True
 
-    def __init__(self):
+    def __init__(self, cooker):
         self._idlefuns = {}
-        self.commands = BitBakeServerCommands(self)
-
-    def addcooker(self, cooker):
-        self.cooker = cooker
-        self.commands.cooker = cooker
+        self.commands = BitBakeServerCommands(self, cooker)
 
     def register_idle_function(self, function, data):
         """Register a function to be called while the server is idle"""
@@ -147,9 +144,7 @@ class BitBakeNoneServer():
                 pass
         if nextsleep is not None:
             #print "Sleeping for %s (%s)" % (nextsleep, delay)
-            signal.signal(signal.SIGCHLD, chldhandler)
             time.sleep(nextsleep)
-            signal.signal(signal.SIGCHLD, signal.SIG_DFL)
 
     def server_exit(self):
         # Tell idle functions we're exiting
@@ -159,13 +154,21 @@ class BitBakeNoneServer():
             except:
                 pass
 
-class BitBakeServerConnection():
+class BitbakeServerInfo():
     def __init__(self, server):
-        self.server = server.server
-        self.connection = self.server.commands
+        self.server = server
+        self.commands = server.commands
+
+class BitBakeServerFork():
+    def __init__(self, serverinfo, command, logfile):
+        serverinfo.forkCommand = command
+        serverinfo.logfile = logfile
+
+class BitBakeServerConnection():
+    def __init__(self, serverinfo):
+        self.server = serverinfo.server
+        self.connection = serverinfo.commands
         self.events = bb.server.none.BBUIEventQueue(self.server)
-        for event in bb.event.ui_queue:
-            self.events.queue_event(event)
 
     def terminate(self):
         try:
@@ -176,28 +179,3 @@ class BitBakeServerConnection():
             self.connection.terminateServer()
         except:
             pass
-
-class BitBakeServer(object):
-    def initServer(self):
-        self.server = BitBakeNoneServer()
-
-    def addcooker(self, cooker):
-        self.cooker = cooker
-        self.server.addcooker(cooker)
-
-    def getServerIdleCB(self):
-        return self.server.register_idle_function
-
-    def saveConnectionDetails(self):
-        return
-
-    def detach(self, cooker_logfile):
-        self.logfile = cooker_logfile
-
-    def establishConnection(self):
-        self.connection = BitBakeServerConnection(self)
-        return self.connection
-
-    def launchUI(self, uifunc, *args):
-        return bb.cooker.server_main(self.cooker, uifunc, *args)
-

bitbake/lib/bb/server/process.py

@@ -1,270 +0,0 @@
-#
-# BitBake Process based server.
-#
-# Copyright (C) 2010 Bob Foerster <robert@erafx.com>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-"""
-    This module implements a multiprocessing.Process based server for bitbake.
-"""
-
-import bb
-import bb.event
-import itertools
-import logging
-import multiprocessing
-import os
-import signal
-import sys
-import time
-from Queue import Empty
-from multiprocessing import Event, Process, util, Queue, Pipe, queues
-
-logger = logging.getLogger('BitBake')
-
-class ServerCommunicator():
-    def __init__(self, connection):
-        self.connection = connection
-
-    def runCommand(self, command):
-        # @todo try/except
-        self.connection.send(command)
-
-        while True:
-            # don't let the user ctrl-c while we're waiting for a response
-            try:
-                if self.connection.poll(.5):
-                    return self.connection.recv()
-                else:
-                    return None
-            except KeyboardInterrupt:
-                pass
-
-
-class EventAdapter():
-    """
-    Adapter to wrap our event queue since the caller (bb.event) expects to
-    call a send() method, but our actual queue only has put()
-    """
-    def __init__(self, queue):
-        self.queue = queue
-
-    def send(self, event):
-        try:
-            self.queue.put(event)
-        except Exception as err:
-            print("EventAdapter puked: %s" % str(err))
-
-
-class ProcessServer(Process):
-    profile_filename = "profile.log"
-    profile_processed_filename = "profile.log.processed"
-
-    def __init__(self, command_channel, event_queue):
-        Process.__init__(self)
-        self.command_channel = command_channel
-        self.event_queue = event_queue
-        self.event = EventAdapter(event_queue)
-        self._idlefunctions = {}
-        self.quit = False
-
-        self.keep_running = Event()
-        self.keep_running.set()
-
-    def register_idle_function(self, function, data):
-        """Register a function to be called while the server is idle"""
-        assert hasattr(function, '__call__')
-        self._idlefunctions[function] = data
-
-    def run(self):
-        for event in bb.event.ui_queue:
-            self.event_queue.put(event)
-        self.event_handle = bb.event.register_UIHhandler(self)
-        bb.cooker.server_main(self.cooker, self.main)
-
-    def main(self):
-        # Ignore SIGINT within the server, as all SIGINT handling is done by
-        # the UI and communicated to us
-        signal.signal(signal.SIGINT, signal.SIG_IGN)
-        while self.keep_running.is_set():
-            try:
-                if self.command_channel.poll():
-                    command = self.command_channel.recv()
-                    self.runCommand(command)
-
-                self.idle_commands(.1)
-            except Exception:
-                logger.exception('Running command %s', command)
-
-        self.event_queue.cancel_join_thread()
-        bb.event.unregister_UIHhandler(self.event_handle)
-        self.command_channel.close()
-        self.cooker.stop()
-        self.idle_commands(.1)
-
-    def idle_commands(self, delay):
-        nextsleep = delay
-
-        for function, data in self._idlefunctions.items():
-            try:
-                retval = function(self, data, False)
-                if retval is False:
-                    del self._idlefunctions[function]
-                elif retval is True:
-                    nextsleep = None
-                elif nextsleep is None:
-                    continue
-                elif retval < nextsleep:
-                    nextsleep = retval
-            except SystemExit:
-                raise
-            except Exception:
-                logger.exception('Running idle function')
-
-        if nextsleep is not None:
-            time.sleep(nextsleep)
-
-    def runCommand(self, command):
-        """
-        Run a cooker command on the server
-        """
-        self.command_channel.send(self.cooker.command.runCommand(command))
-
-    def stop(self):
-        self.keep_running.clear()
-
-    def bootstrap_2_6_6(self):
-        """Pulled from python 2.6.6. Needed to ensure we have the fix from
-        http://bugs.python.org/issue5313 when running on python version 2.6.2
-        or lower."""
-
-        try:
-            self._children = set()
-            self._counter = itertools.count(1)
-            try:
-                sys.stdin.close()
-                sys.stdin = open(os.devnull)
-            except (OSError, ValueError):
-                pass
-            multiprocessing._current_process = self
-            util._finalizer_registry.clear()
-            util._run_after_forkers()
-            util.info('child process calling self.run()')
-            try:
-                self.run()
-                exitcode = 0
-            finally:
-                util._exit_function()
-        except SystemExit as e:
-            if not e.args:
-                exitcode = 1
-            elif type(e.args[0]) is int:
-                exitcode = e.args[0]
-            else:
-                sys.stderr.write(e.args[0] + '\n')
-                sys.stderr.flush()
-                exitcode = 1
-        except:
-            exitcode = 1
-            import traceback
-            sys.stderr.write('Process %s:\n' % self.name)
-            sys.stderr.flush()
-            traceback.print_exc()
-
-        util.info('process exiting with exitcode %d' % exitcode)
-        return exitcode
-
-    # Python versions 2.6.0 through 2.6.2 suffer from a multiprocessing bug
-    # which can result in a bitbake server hang during the parsing process
-    if (2, 6, 0) <= sys.version_info < (2, 6, 3):
-        _bootstrap = bootstrap_2_6_6
-
-class BitBakeServerConnection():
-    def __init__(self, server):
-        self.server = server
-        self.procserver = server.server
-        self.connection = ServerCommunicator(server.ui_channel)
-        self.events = server.event_queue
-
-    def terminate(self, force = False):
-        signal.signal(signal.SIGINT, signal.SIG_IGN)
-        self.procserver.stop()
-        if force:
-            self.procserver.join(0.5)
-            if self.procserver.is_alive():
-                self.procserver.terminate()
-                self.procserver.join()
-        else:
-            self.procserver.join()
-        while True:
-            try:
-                event = self.server.event_queue.get(block=False)
-            except (Empty, IOError):
-                break
-            if isinstance(event, logging.LogRecord):
-                logger.handle(event)
-        self.server.ui_channel.close()
-        self.server.event_queue.close()
-        if force:
-            sys.exit(1)
-
-# Wrap Queue to provide API which isn't server implementation specific
-class ProcessEventQueue(multiprocessing.queues.Queue):
-    def waitEvent(self, timeout):
-        try:
-            return self.get(True, timeout)
-        except Empty:
-            return None
-
-    def getEvent(self):
-        try:
-            return self.get(False)
-        except Empty:
-            return None
-
-
-class BitBakeServer(object):
-    def initServer(self):
-        # establish communication channels.  We use bidirectional pipes for
-        # ui <--> server command/response pairs
-        # and a queue for server -> ui event notifications
-        #
-        self.ui_channel, self.server_channel = Pipe()
-        self.event_queue = ProcessEventQueue(0)
-
-        self.server = ProcessServer(self.server_channel, self.event_queue)
-
-    def addcooker(self, cooker):
-        self.cooker = cooker
-        self.server.cooker = cooker
-
-    def getServerIdleCB(self):
-        return self.server.register_idle_function
-
-    def saveConnectionDetails(self):
-        return
-
-    def detach(self, cooker_logfile):
-        self.server.start() 
-        return
-
-    def establishConnection(self):
-        self.connection = BitBakeServerConnection(self)
-        signal.signal(signal.SIGTERM, lambda i, s: self.connection.terminate(force=True))
-        return self.connection
-
-    def launchUI(self, uifunc, *args):
-        return bb.cooker.server_main(self.cooker, uifunc, *args)
-

bitbake/lib/bb/server/xmlrpc.py

@@ -45,92 +45,16 @@ if sys.hexversion < 0x020600F0:
     print("Sorry, python 2.6 or later is required for bitbake's XMLRPC mode")
     sys.exit(1)
 
-##
-# The xmlrpclib.Transport class has undergone various changes in Python 2.7
-# which break BitBake's XMLRPC implementation.
-# To work around this we subclass Transport and have a copy/paste of method
-# implementations from Python 2.6.6's xmlrpclib.
-#
-# Upstream Python bug is #8194 (http://bugs.python.org/issue8194)
-# This bug is relevant for Python 2.7.0 and 2.7.1 but was fixed for
-# Python > 2.7.2
-##
-
-class BBTransport(xmlrpclib.Transport):
-    def request(self, host, handler, request_body, verbose=0):
-        h = self.make_connection(host)
-        if verbose:
-            h.set_debuglevel(1)
-
-        self.send_request(h, handler, request_body)
-        self.send_host(h, host)
-        self.send_user_agent(h)
-        self.send_content(h, request_body)
-
-        errcode, errmsg, headers = h.getreply()
-
-        if errcode != 200:
-            raise ProtocolError(
-                host + handler,
-                errcode, errmsg,
-                headers
-                )
-
-        self.verbose = verbose
-
-        try:
-            sock = h._conn.sock
-        except AttributeError:
-            sock = None
-
-        return self._parse_response(h.getfile(), sock)
-
-    def make_connection(self, host):
-        import httplib
-        host, extra_headers, x509 = self.get_host_info(host)
-        return httplib.HTTP(host)
-
-    def _parse_response(self, file, sock):
-        p, u = self.getparser()
-
-        while 1:
-            if sock:
-                response = sock.recv(1024)
-            else:
-                response = file.read(1024)
-            if not response:
-                break
-            if self.verbose:
-                print "body:", repr(response)
-            p.feed(response)
-
-        file.close()
-        p.close()
-
-        return u.close()
-
-def _create_server(host, port):
-    # Python 2.7.0 and 2.7.1 have a buggy Transport implementation
-    # For those versions of Python, and only those versions, use our
-    # own copy/paste BBTransport class.
-    if (2, 7, 0) <= sys.version_info < (2, 7, 2):
-        t = BBTransport()
-        s = xmlrpclib.Server("http://%s:%d/" % (host, port), transport=t, allow_none=True)
-    else:
-        s = xmlrpclib.Server("http://%s:%d/" % (host, port), allow_none=True)
-
-    return s
-
 class BitBakeServerCommands():
-    def __init__(self, server):
+    def __init__(self, server, cooker):
+        self.cooker = cooker
         self.server = server
 
     def registerEventHandler(self, host, port):
         """
         Register a remote UI Event Handler
         """
-        s = _create_server(host, port)
-
+        s = xmlrpclib.Server("http://%s:%d" % (host, port), allow_none=True)
         return bb.event.register_UIHhandler(s)
 
     def unregisterEventHandler(self, handlerNum):
@@ -150,7 +74,7 @@ class BitBakeServerCommands():
         Trigger the server to quit
         """
         self.server.quit = True
-        print("Server (cooker) exiting")
+        print("Server (cooker) exitting")
         return
 
     def ping(self):
@@ -159,11 +83,11 @@ class BitBakeServerCommands():
         """
         return True
 
-class BitBakeXMLRPCServer(SimpleXMLRPCServer):
+class BitBakeServer(SimpleXMLRPCServer):
     # remove this when you're done with debugging
     # allow_reuse_address = True
 
-    def __init__(self, interface = ("localhost", 0)):
+    def __init__(self, cooker, interface = ("localhost", 0)):
         """
         Constructor
         """
@@ -173,12 +97,8 @@ class BitBakeXMLRPCServer(SimpleXMLRPCServer):
         self._idlefuns = {}
         self.host, self.port = self.socket.getsockname()
         #self.register_introspection_functions()
-        self.commands = BitBakeServerCommands(self)
-        self.autoregister_all_functions(self.commands, "")
-
-    def addcooker(self, cooker):
-        self.cooker = cooker
-        self.commands.cooker = cooker
+        commands = BitBakeServerCommands(self, cooker)
+        self.autoregister_all_functions(commands, "")
 
     def autoregister_all_functions(self, context, prefix):
         """
@@ -196,9 +116,6 @@ class BitBakeXMLRPCServer(SimpleXMLRPCServer):
         self._idlefuns[function] = data
 
     def serve_forever(self):
-        bb.cooker.server_main(self.cooker, self._serve_forever)
-
-    def _serve_forever(self):
         """
         Serve Requests. Overloaded to honor a quit command
         """
@@ -246,12 +163,14 @@ class BitbakeServerInfo():
         self.host = server.host
         self.port = server.port
 
+class BitBakeServerFork():
+    def __init__(self, serverinfo, command, logfile):
+        daemonize.createDaemon(command, logfile)
+
 class BitBakeServerConnection():
     def __init__(self, serverinfo):
-        self.connection = _create_server(serverinfo.host, serverinfo.port)
+        self.connection = xmlrpclib.Server("http://%s:%s" % (serverinfo.host, serverinfo.port),  allow_none=True)
         self.events = uievent.BBUIEventQueue(self.connection)
-        for event in bb.event.ui_queue:
-            self.events.queue_event(event)
 
     def terminate(self):
         # Don't wait for server indefinitely
@@ -265,31 +184,3 @@ class BitBakeServerConnection():
             self.connection.terminateServer()
         except:
             pass
-
-class BitBakeServer(object):
-    def initServer(self):
-        self.server = BitBakeXMLRPCServer()
-
-    def addcooker(self, cooker):
-        self.cooker = cooker
-        self.server.addcooker(cooker)
-
-    def getServerIdleCB(self):
-        return self.server.register_idle_function
-
-    def saveConnectionDetails(self): 
-        self.serverinfo = BitbakeServerInfo(self.server)
-
-    def detach(self, cooker_logfile):
-        daemonize.createDaemon(self.server.serve_forever, cooker_logfile)
-        del self.cooker
-        del self.server
-
-    def establishConnection(self):
-        self.connection = BitBakeServerConnection(self.serverinfo)
-        return self.connection
-
-    def launchUI(self, uifunc, *args):
-        return uifunc(*args)
-
-

bitbake/lib/bb/shell.py

@@ -180,9 +180,11 @@ class BitBakeShellCommands:
             last_exception = Providers.NoProvider
 
         except runqueue.TaskFailure as fnids:
+            for fnid in fnids:
+                print("ERROR: '%s' failed" % td.fn_index[fnid])
             last_exception = runqueue.TaskFailure
 
-        except build.FuncFailed as e:
+        except build.EventException as e:
             print("ERROR: Couldn't build '%s'" % names)
             last_exception = e
 
@@ -245,7 +247,7 @@ class BitBakeShellCommands:
             cooker.buildFile(bf, cmd)
         except parse.ParseError:
             print("ERROR: Unable to open or parse '%s'" % bf)
-        except build.FuncFailed as e:
+        except build.EventException as e:
             print("ERROR: Couldn't build '%s'" % name)
             last_exception = e
 
@@ -272,7 +274,9 @@ class BitBakeShellCommands:
         bbfile = params[0]
         print("SHELL: Parsing '%s'" % bbfile)
         parse.update_mtime( bbfile )
-        cooker.parser.reparse(bbfile)
+        cooker.bb_cache.cacheValidUpdate(bbfile)
+        fromCache = cooker.bb_cache.loadData(bbfile, cooker.configuration.data, cooker.status)
+        cooker.bb_cache.sync()
         if False: #fromCache:
             print("SHELL: File has not been updated, not reparsing")
         else:
@@ -407,7 +411,7 @@ SRC_URI = ""
 
     def parse( self, params ):
         """(Re-)parse .bb files and calculate the dependency graph"""
-        cooker.status = cache.CacheData(cooker.caches_array)
+        cooker.status = cache.CacheData()
         ignore = data.getVar("ASSUME_PROVIDED", cooker.configuration.data, 1) or ""
         cooker.status.ignored_dependencies = set( ignore.split() )
         cooker.handleCollections( data.getVar("BBFILE_COLLECTIONS", cooker.configuration.data, 1) )
@@ -441,7 +445,7 @@ SRC_URI = ""
         name, var = params
         bbfile = self._findProvider( name )
         if bbfile is not None:
-            the_data = cache.Cache.loadDataFull(bbfile, cooker.configuration.data)
+            the_data = cooker.bb_cache.loadDataFull(bbfile, cooker.configuration.data)
             value = the_data.getVar( var, 1 )
             print(value)
         else:

bitbake/lib/bb/siggen.py

@@ -1,53 +1,35 @@
 import hashlib
-import logging
-import os
 import re
-import bb.data
-
-logger = logging.getLogger('BitBake.SigGen')
 
 try:
     import cPickle as pickle
 except ImportError:
     import pickle
-    logger.info('Importing cPickle failed.  Falling back to a very slow implementation.')
+    bb.msg.note(1, bb.msg.domain.Cache, "Importing cPickle failed. Falling back to a very slow implementation.")
 
-def init(d):
+def init(d, dumpsigs):
     siggens = [obj for obj in globals().itervalues()
                       if type(obj) is type and issubclass(obj, SignatureGenerator)]
 
     desired = bb.data.getVar("BB_SIGNATURE_HANDLER", d, True) or "noop"
     for sg in siggens:
         if desired == sg.name:
-            return sg(d)
+            return sg(d, dumpsigs)
             break
     else:
-        logger.error("Invalid signature generator '%s', using default 'noop'\n"
-                     "Available generators: %s",
-                     ', '.join(obj.name for obj in siggens))
-        return SignatureGenerator(d)
+        bb.error("Invalid signature generator '%s', using default 'noop' generator" % desired)
+        bb.error("Available generators: %s" % ", ".join(obj.name for obj in siggens))
+        return SignatureGenerator(d, dumpsigs)
 
 class SignatureGenerator(object):
     """
     """
     name = "noop"
 
-    def __init__(self, data):
+    def __init__(self, data, dumpsigs):
         return
 
-    def finalise(self, fn, d, varient):
-        return
-
-    def get_taskhash(self, fn, task, deps, dataCache):
-        return "0"
-
-    def set_taskdata(self, hashes, deps):
-        return
-
-    def stampfile(self, stampbase, file_name, taskname, extrainfo):
-        return ("%s.%s.%s" % (stampbase, taskname, extrainfo)).rstrip('.')
-
-    def dump_sigtask(self, fn, task, stampbase, runtime):
+    def finalise(self, fn, d):
         return
 
 class SignatureGeneratorBasic(SignatureGenerator):
@@ -55,14 +37,14 @@ class SignatureGeneratorBasic(SignatureGenerator):
     """
     name = "basic"
 
-    def __init__(self, data):
+    def __init__(self, data, dumpsigs):
         self.basehash = {}
         self.taskhash = {}
         self.taskdeps = {}
         self.runtaskdeps = {}
         self.gendeps = {}
         self.lookupcache = {}
-        self.basewhitelist = set((data.getVar("BB_HASHBASE_WHITELIST", True) or "").split())
+        self.basewhitelist = (data.getVar("BB_HASHBASE_WHITELIST", True) or "").split()
         self.taskwhitelist = data.getVar("BB_HASHTASK_WHITELIST", True) or None
 
         if self.taskwhitelist:
@@ -70,50 +52,35 @@ class SignatureGeneratorBasic(SignatureGenerator):
         else:
             self.twl = None
 
+        self.dumpsigs = dumpsigs
+
     def _build_data(self, fn, d):
 
-        tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d)
+        taskdeps, gendeps = bb.data.generate_dependencies(d)
 
-        taskdeps = {}
         basehash = {}
+        lookupcache = {}
 
-        for task in tasklist:
+        for task in taskdeps:
             data = d.getVar(task, False)
             lookupcache[task] = data
-
-            if data is None:
-                bb.error("Task %s from %s seems to be empty?!" % (task, fn))
-                data = ''
-
-            newdeps = gendeps[task]
-            seen = set()
-            while newdeps:
-                nextdeps = newdeps
-                seen |= nextdeps
-                newdeps = set()
-                for dep in nextdeps:
-                    if dep in self.basewhitelist:
-                        continue
-                    newdeps |= gendeps[dep]
-                newdeps -= seen
-
-            alldeps = seen - self.basewhitelist
-
-            for dep in sorted(alldeps):
-                data = data + dep
+            for dep in sorted(taskdeps[task]):
+                if dep in self.basewhitelist:
+                    continue
                 if dep in lookupcache:
                     var = lookupcache[dep]
                 else:
                     var = d.getVar(dep, False)
                     lookupcache[dep] = var
                 if var:
-                    data = data + str(var)
+                    data = data + var
             self.basehash[fn + "." + task] = hashlib.md5(data).hexdigest()
-            taskdeps[task] = sorted(alldeps)
+            #bb.note("Hash for %s is %s" % (task, tashhash[task]))
 
-        self.taskdeps[fn] = taskdeps
-        self.gendeps[fn] = gendeps
-        self.lookupcache[fn] = lookupcache
+        if self.dumpsigs:
+            self.taskdeps[fn] = taskdeps
+            self.gendeps[fn] = gendeps
+            self.lookupcache[fn] = lookupcache
 
         return taskdeps
 
@@ -134,18 +101,14 @@ class SignatureGeneratorBasic(SignatureGenerator):
     def get_taskhash(self, fn, task, deps, dataCache):
         k = fn + "." + task
         data = dataCache.basetaskhash[k]
-        self.runtaskdeps[k] = []
-        for dep in sorted(deps, key=clean_basepath):
-            # We only manipulate the dependencies for packages not in the whitelist
-            if self.twl and not self.twl.search(dataCache.pkg_fn[fn]):
-                # then process the actual dependencies
-                dep_fn = re.search("(?P<fn>.*)\..*", dep).group('fn')
-                if self.twl.search(dataCache.pkg_fn[dep_fn]):
-                    continue
+        self.runtaskdeps[k] = deps
+        for dep in sorted(deps):
+            if self.twl and self.twl.search(dataCache.pkg_fn[fn]):
+                #bb.note("Skipping %s" % dep)
+                continue
             if dep not in self.taskhash:
-                bb.fatal("%s is not in taskhash, caller isn't calling in dependency order?", dep)
+                 bb.fatal("%s is not in taskhash, caller isn't calling in dependency order?", dep)
             data = data + self.taskhash[dep]
-            self.runtaskdeps[k].append(dep)
         h = hashlib.md5(data).hexdigest()
         self.taskhash[k] = h
         #d.setVar("BB_TASKHASH_task-%s" % task, taskhash[task])
@@ -159,13 +122,11 @@ class SignatureGeneratorBasic(SignatureGenerator):
         k = fn + "." + task
         if runtime == "customfile":
             sigfile = stampbase
-        elif runtime and k in self.taskhash:
+        elif runtime:
             sigfile = stampbase + "." + task + ".sigdata" + "." + self.taskhash[k]
         else:
             sigfile = stampbase + "." + task + ".sigbasedata" + "." + self.basehash[k]
 
-        bb.utils.mkdirhier(os.path.dirname(sigfile))
-
         data = {}
         data['basewhitelist'] = self.basewhitelist
         data['taskwhitelist'] = self.taskwhitelist
@@ -180,7 +141,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
             data['gendeps'][dep] = self.gendeps[fn][dep]
             data['varvals'][dep] = self.lookupcache[fn][dep]
 
-        if runtime and k in self.taskhash:
+        if runtime:
             data['runtaskdeps'] = self.runtaskdeps[k]
             data['runtaskhashes'] = {}
             for dep in data['runtaskdeps']:
@@ -196,77 +157,53 @@ class SignatureGeneratorBasic(SignatureGenerator):
                 if k not in self.taskhash:
                     continue
                 if dataCache.basetaskhash[k] != self.basehash[k]:
-                    bb.error("Bitbake's cached basehash does not match the one we just generated (%s)!" % k)
+                    bb.error("Bitbake's cached basehash does not match the one we just generated!")
                     bb.error("The mismatched hashes were %s and %s" % (dataCache.basetaskhash[k], self.basehash[k]))
                 self.dump_sigtask(fn, task, dataCache.stamp[fn], True)
 
-class SignatureGeneratorBasicHash(SignatureGeneratorBasic):
-    name = "basichash"
-
-    def stampfile(self, stampbase, fn, taskname, extrainfo):
-        if taskname != "do_setscene" and taskname.endswith("_setscene"):
-            k = fn + "." + taskname[:-9]
-        else:
-            k = fn + "." + taskname
-        h = self.taskhash[k]
-        return ("%s.%s.%s.%s" % (stampbase, taskname, h, extrainfo)).rstrip('.')
-
 def dump_this_task(outfile, d):
-    import bb.parse
     fn = d.getVar("BB_FILENAME", True)
     task = "do_" + d.getVar("BB_CURRENTTASK", True)
     bb.parse.siggen.dump_sigtask(fn, task, outfile, "customfile")
 
-def clean_basepath(a):
-    if a.startswith("virtual:"):
-        b = a.rsplit(":", 1)[0] + a.rsplit("/", 1)[1]
-    else:
-        b = a.rsplit("/", 1)[1]
-    return b
-
-def clean_basepaths(a):
-    b = {}
-    for x in a:
-        b[clean_basepath(x)] = a[x]
-    return b
-
 def compare_sigfiles(a, b):
     p1 = pickle.Unpickler(file(a, "rb"))
     a_data = p1.load()
     p2 = pickle.Unpickler(file(b, "rb"))
     b_data = p2.load()
 
-    def dict_diff(a, b, whitelist=set()):
+    #print "Checking"
+    #print str(a_data)
+    #print str(b_data)
+
+    def dict_diff(a, b):
         sa = set(a.keys())
         sb = set(b.keys())
         common = sa & sb
         changed = set()
         for i in common:
-            if a[i] != b[i] and i not in whitelist:
+            if a[i] != b[i]:
                 changed.add(i)
         added = sa - sb
         removed = sb - sa
-        return changed, added, removed
+        return changed, added, removed 
 
     if 'basewhitelist' in a_data and a_data['basewhitelist'] != b_data['basewhitelist']:
         print "basewhitelist changed from %s to %s" % (a_data['basewhitelist'], b_data['basewhitelist'])
-        print "changed items: %s" % a_data['basewhitelist'].symmetric_difference(b_data['basewhitelist'])
 
     if 'taskwhitelist' in a_data and a_data['taskwhitelist'] != b_data['taskwhitelist']:
         print "taskwhitelist changed from %s to %s" % (a_data['taskwhitelist'], b_data['taskwhitelist'])
-        print "changed items: %s" % a_data['taskwhitelist'].symmetric_difference(b_data['taskwhitelist'])
 
     if a_data['taskdeps'] != b_data['taskdeps']:
-        print "Task dependencies changed from:\n%s\nto:\n%s" % (sorted(a_data['taskdeps']), sorted(b_data['taskdeps']))
+        print "Task dependencies changed from %s to %s" % (sorted(a_data['taskdeps']), sorted(b_data['taskdeps']))
 
     if a_data['basehash'] != b_data['basehash']:
         print "basehash changed from %s to %s" % (a_data['basehash'], b_data['basehash'])
 
-    changed, added, removed = dict_diff(a_data['gendeps'], b_data['gendeps'], a_data['basewhitelist'] & b_data['basewhitelist'])
+    changed, added, removed = dict_diff(a_data['gendeps'], b_data['gendeps'])
     if changed:
         for dep in changed:
             print "List of dependencies for variable %s changed from %s to %s" % (dep, a_data['gendeps'][dep], b_data['gendeps'][dep])
-            print "changed items: %s" % a_data['gendeps'][dep].symmetric_difference(b_data['gendeps'][dep])
     if added:
         for dep in added:
             print "Dependency on variable %s was added" % (dep)
@@ -279,23 +216,18 @@ def compare_sigfiles(a, b):
     if changed:
         for dep in changed:
             print "Variable %s value changed from %s to %s" % (dep, a_data['varvals'][dep], b_data['varvals'][dep])
+    #if added:
+    #    print "Dependency on variable %s was added (value %s)" % (dep, b_data['gendeps'][dep])
+    #if removed:
+    #    print "Dependency on Variable %s was removed (value %s)" % (dep, a_data['gendeps'][dep])
 
-    if 'runtaskhashes' in a_data and 'runtaskhashes' in b_data:
-        a = clean_basepaths(a_data['runtaskhashes'])
-        b = clean_basepaths(b_data['runtaskhashes'])
-        changed, added, removed = dict_diff(a, b)
-        if added:
-            for dep in added:
-                print "Dependency on task %s was added" % (dep)
-        if removed:
-            for dep in removed:
-                print "Dependency on task %s was removed" % (dep)
-        if changed:
-            for dep in changed:
-                print "Hash for dependent task %s changed from %s to %s" % (dep, a[dep], b[dep])
-    elif 'runtaskdeps' in a_data and 'runtaskdeps' in b_data and sorted(a_data['runtaskdeps']) != sorted(b_data['runtaskdeps']):
-        print "Tasks this task depends on changed from %s to %s" % (sorted(a_data['runtaskdeps']), sorted(b_data['runtaskdeps']))
-        print "changed items: %s" % a_data['runtaskdeps'].symmetric_difference(b_data['runtaskdeps'])
+    if 'runtaskdeps' in a_data and 'runtaskdeps' in b_data and a_data['runtaskdeps'] != b_data['runtaskdeps']:
+        print "Tasks this task depends on changed from %s to %s" % (a_data['taskdeps'], b_data['taskdeps'])
+
+    if 'runtaskhashes' in a_data:
+        for dep in a_data['runtaskhashes']:
+            if a_data['runtaskhashes'][dep] != b_data['runtaskhashes'][dep]:
+                print "Hash for dependent task %s changed from %s to %s" % (dep, a_data['runtaskhashes'][dep], b_data['runtaskhashes'][dep])
 
 def dump_sigfile(a):
     p1 = pickle.Unpickler(file(a, "rb"))
@@ -321,3 +253,8 @@ def dump_sigfile(a):
     if 'runtaskhashes' in a_data:
         for dep in a_data['runtaskhashes']:
             print "Hash for dependent task %s is %s" % (dep, a_data['runtaskhashes'][dep])
+
+
+
+
+

bitbake/lib/bb/taskdata.py

@@ -23,25 +23,26 @@ Task data collection and handling
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
-import logging
-import re
 import bb
 
-logger = logging.getLogger("BitBake.TaskData")
-
 def re_match_strings(target, strings):
     """
     Whether or not the string 'target' matches
     any one string of the strings which can be regular expression string
     """
-    return any(name == target or re.match(name, target)
-               for name in strings)
+    import re
+
+    for name in strings:
+        if (name==target or
+                re.search(name, target)!=None):
+            return True
+    return False
 
 class TaskData:
     """
     BitBake Task Data implementation
     """
-    def __init__(self, abort = True, tryaltconfigs = False, skiplist = None):
+    def __init__(self, abort = True, tryaltconfigs = False):
         self.build_names_index = []
         self.run_names_index = []
         self.fn_index = []
@@ -70,8 +71,6 @@ class TaskData:
         self.abort = abort
         self.tryaltconfigs = tryaltconfigs
 
-        self.skiplist = skiplist
-
     def getbuild_id(self, name):
         """
         Return an ID number for the build target name.
@@ -153,7 +152,7 @@ class TaskData:
         fnid = self.getfn_id(fn)
 
         if fnid in self.failed_fnids:
-            bb.msg.fatal("TaskData", "Trying to re-add a failed file? Something is broken...")
+            bb.msg.fatal(bb.msg.domain.TaskData, "Trying to re-add a failed file? Something is broken...")
 
         # Check if we've already seen this fn
         if fnid in self.tasks_fnid:
@@ -175,7 +174,7 @@ class TaskData:
                 for dep in task_deps['depends'][task].split():
                     if dep:
                         if ":" not in dep:
-                            bb.msg.fatal("TaskData", "Error for %s, dependency %s does not contain ':' character\n. Task 'depends' should be specified in the form 'packagename:task'" % (fn, dep))
+                            bb.msg.fatal(bb.msg.domain.TaskData, "Error, dependency %s does not contain ':' character\n. Task 'depends' should be specified in the form 'packagename:task'" % (dep, fn))
                         ids.append(((self.getbuild_id(dep.split(":")[0])), dep.split(":")[1]))
                 self.tasks_idepends[taskid].extend(ids)
 
@@ -183,7 +182,7 @@ class TaskData:
         if not fnid in self.depids:
             dependids = {}
             for depend in dataCache.deps[fn]:
-                logger.debug(2, "Added dependency %s for %s", depend, fn)
+                bb.msg.debug(2, bb.msg.domain.TaskData, "Added dependency %s for %s" % (depend, fn))
                 dependids[self.getbuild_id(depend)] = None
             self.depids[fnid] = dependids.keys()
 
@@ -193,12 +192,12 @@ class TaskData:
             rdepends = dataCache.rundeps[fn]
             rrecs = dataCache.runrecs[fn]
             for package in rdepends:
-                for rdepend in rdepends[package]:
-                    logger.debug(2, "Added runtime dependency %s for %s", rdepend, fn)
+                for rdepend in bb.utils.explode_deps(rdepends[package]):
+                    bb.msg.debug(2, bb.msg.domain.TaskData, "Added runtime dependency %s for %s" % (rdepend, fn))
                     rdependids[self.getrun_id(rdepend)] = None
             for package in rrecs:
-                for rdepend in rrecs[package]:
-                    logger.debug(2, "Added runtime recommendation %s for %s", rdepend, fn)
+                for rdepend in bb.utils.explode_deps(rrecs[package]):
+                    bb.msg.debug(2, bb.msg.domain.TaskData, "Added runtime recommendation %s for %s" % (rdepend, fn))
                     rdependids[self.getrun_id(rdepend)] = None
             self.rdepids[fnid] = rdependids.keys()
 
@@ -350,22 +349,6 @@ class TaskData:
                 dependees.append(self.fn_index[fnid])
         return dependees
 
-    def get_reasons(self, item, runtime=False):
-        """
-        Get the reason(s) for an item not being provided, if any
-        """
-        reasons = []
-        if self.skiplist:
-            for fn in self.skiplist:
-                skipitem = self.skiplist[fn]
-                if skipitem.pn == item:
-                    reasons.append("%s was skipped: %s" % (skipitem.pn, skipitem.skipreason))
-                elif runtime and item in skipitem.rprovides:
-                    reasons.append("%s RPROVIDES %s but was skipped: %s" % (skipitem.pn, item, skipitem.skipreason))
-                elif not runtime and item in skipitem.provides:
-                    reasons.append("%s PROVIDES %s but was skipped: %s" % (skipitem.pn, item, skipitem.skipreason))
-        return reasons
-
     def add_provider(self, cfgData, dataCache, item):
         try:
             self.add_provider_internal(cfgData, dataCache, item)
@@ -387,7 +370,7 @@ class TaskData:
             return
 
         if not item in dataCache.providers:
-            bb.event.fire(bb.event.NoProvider(item, dependees=self.get_dependees_str(item), reasons=self.get_reasons(item)), cfgData)
+            bb.event.fire(bb.event.NoProvider(item, dependees=self.get_rdependees_str(item)), cfgData)
             raise bb.providers.NoProvider(item)
 
         if self.have_build_target(item):
@@ -399,7 +382,7 @@ class TaskData:
         eligible = [p for p in eligible if not self.getfn_id(p) in self.failed_fnids]
 
         if not eligible:
-            bb.event.fire(bb.event.NoProvider(item, dependees=self.get_dependees_str(item), reasons=["No eligible PROVIDERs exist for '%s'" % item]), cfgData)
+            bb.event.fire(bb.event.NoProvider(item, dependees=self.get_dependees_str(item)), cfgData)
             raise bb.providers.NoProvider(item)
 
         if len(eligible) > 1 and foundUnique == False:
@@ -414,7 +397,7 @@ class TaskData:
             fnid = self.getfn_id(fn)
             if fnid in self.failed_fnids:
                 continue
-            logger.debug(2, "adding %s to satisfy %s", fn, item)
+            bb.msg.debug(2, bb.msg.domain.Provider, "adding %s to satisfy %s" % (fn, item))
             self.add_build_target(fn, item)
             self.add_tasks(fn, dataCache)
 
@@ -436,14 +419,14 @@ class TaskData:
         all_p = bb.providers.getRuntimeProviders(dataCache, item)
 
         if not all_p:
-            bb.event.fire(bb.event.NoProvider(item, runtime=True, dependees=self.get_rdependees_str(item), reasons=self.get_reasons(item, True)), cfgData)
+            bb.event.fire(bb.event.NoProvider(item, runtime=True, dependees=self.get_rdependees_str(item)), cfgData)
             raise bb.providers.NoRProvider(item)
 
         eligible, numberPreferred = bb.providers.filterProvidersRunTime(all_p, item, cfgData, dataCache)
         eligible = [p for p in eligible if not self.getfn_id(p) in self.failed_fnids]
 
         if not eligible:
-            bb.event.fire(bb.event.NoProvider(item, runtime=True, dependees=self.get_rdependees_str(item), reasons=["No eligible RPROVIDERs exist for '%s'" % item]), cfgData)
+            bb.event.fire(bb.event.NoProvider(item, runtime=True, dependees=self.get_rdependees_str(item)), cfgData)
             raise bb.providers.NoRProvider(item)
 
         if len(eligible) > 1 and numberPreferred == 0:
@@ -467,7 +450,7 @@ class TaskData:
             fnid = self.getfn_id(fn)
             if fnid in self.failed_fnids:
                 continue
-            logger.debug(2, "adding '%s' to satisfy runtime '%s'", fn, item)
+            bb.msg.debug(2, bb.msg.domain.Provider, "adding '%s' to satisfy runtime '%s'" % (fn, item))
             self.add_runtime_target(fn, item)
             self.add_tasks(fn, dataCache)
 
@@ -480,7 +463,7 @@ class TaskData:
         """
         if fnid in self.failed_fnids:
             return
-        logger.debug(1, "File '%s' is unbuildable, removing...", self.fn_index[fnid])
+        bb.msg.debug(1, bb.msg.domain.Provider, "File '%s' is unbuildable, removing..." % self.fn_index[fnid])
         self.failed_fnids.append(fnid)
         for target in self.build_targets:
             if fnid in self.build_targets[target]:
@@ -502,12 +485,12 @@ class TaskData:
             missing_list = [self.build_names_index[targetid]]
         else:
             missing_list = [self.build_names_index[targetid]] + missing_list
-        logger.verbose("Target '%s' is unbuildable, removing...\nMissing or unbuildable dependency chain was: %s", self.build_names_index[targetid], missing_list)
+        bb.msg.note(2, bb.msg.domain.Provider, "Target '%s' is unbuildable, removing...\nMissing or unbuildable dependency chain was: %s" % (self.build_names_index[targetid], missing_list))
         self.failed_deps.append(targetid)
         dependees = self.get_dependees(targetid)
         for fnid in dependees:
             self.fail_fnid(fnid, missing_list)
-        for taskid in xrange(len(self.tasks_idepends)):
+        for taskid in range(len(self.tasks_idepends)):
             idepends = self.tasks_idepends[taskid]
             for (idependid, idependtask) in idepends:
                 if idependid == targetid:
@@ -515,7 +498,7 @@ class TaskData:
 
         if self.abort and targetid in self.external_targets:
             target = self.build_names_index[targetid]
-            logger.error("Required build target '%s' has no buildable providers.\nMissing or unbuildable dependency chain was: %s", target, missing_list)
+            bb.msg.error(bb.msg.domain.Provider, "Required build target '%s' has no buildable providers.\nMissing or unbuildable dependency chain was: %s" % (target, missing_list))
             raise bb.providers.NoProvider(target)
 
     def remove_runtarget(self, targetid, missing_list = []):
@@ -528,7 +511,7 @@ class TaskData:
         else:
             missing_list = [self.run_names_index[targetid]] + missing_list
 
-        logger.info("Runtime target '%s' is unbuildable, removing...\nMissing or unbuildable dependency chain was: %s", self.run_names_index[targetid], missing_list)
+        bb.msg.note(1, bb.msg.domain.Provider, "Runtime target '%s' is unbuildable, removing...\nMissing or unbuildable dependency chain was: %s" % (self.run_names_index[targetid], missing_list))
         self.failed_rdeps.append(targetid)
         dependees = self.get_rdependees(targetid)
         for fnid in dependees:
@@ -538,7 +521,7 @@ class TaskData:
         """
         Resolve all unresolved build and runtime targets
         """
-        logger.info("Resolving any missing task queue dependencies")
+        bb.msg.note(1, bb.msg.domain.TaskData, "Resolving any missing task queue dependencies")
         while True:
             added = 0
             for target in self.get_unresolved_build_targets(dataCache):
@@ -556,7 +539,7 @@ class TaskData:
                     added = added + 1
                 except bb.providers.NoRProvider:
                     self.remove_runtarget(self.getrun_id(target))
-            logger.debug(1, "Resolved " + str(added) + " extra dependencies")
+            bb.msg.debug(1, bb.msg.domain.TaskData, "Resolved " + str(added) + " extra dependencies")
             if added == 0:
                 break
         # self.dump_data()
@@ -565,40 +548,40 @@ class TaskData:
         """
         Dump some debug information on the internal data structures
         """
-        logger.debug(3, "build_names:")
-        logger.debug(3, ", ".join(self.build_names_index))
+        bb.msg.debug(3, bb.msg.domain.TaskData, "build_names:")
+        bb.msg.debug(3, bb.msg.domain.TaskData, ", ".join(self.build_names_index))
 
-        logger.debug(3, "run_names:")
-        logger.debug(3, ", ".join(self.run_names_index))
+        bb.msg.debug(3, bb.msg.domain.TaskData, "run_names:")
+        bb.msg.debug(3, bb.msg.domain.TaskData, ", ".join(self.run_names_index))
 
-        logger.debug(3, "build_targets:")
-        for buildid in xrange(len(self.build_names_index)):
+        bb.msg.debug(3, bb.msg.domain.TaskData, "build_targets:")
+        for buildid in range(len(self.build_names_index)):
             target = self.build_names_index[buildid]
             targets = "None"
             if buildid in self.build_targets:
                 targets = self.build_targets[buildid]
-            logger.debug(3, " (%s)%s: %s", buildid, target, targets)
+            bb.msg.debug(3, bb.msg.domain.TaskData, " (%s)%s: %s" % (buildid, target, targets))
 
-        logger.debug(3, "run_targets:")
-        for runid in xrange(len(self.run_names_index)):
+        bb.msg.debug(3, bb.msg.domain.TaskData, "run_targets:")
+        for runid in range(len(self.run_names_index)):
             target = self.run_names_index[runid]
             targets = "None"
             if runid in self.run_targets:
                 targets = self.run_targets[runid]
-            logger.debug(3, " (%s)%s: %s", runid, target, targets)
+            bb.msg.debug(3, bb.msg.domain.TaskData, " (%s)%s: %s" % (runid, target, targets))
 
-        logger.debug(3, "tasks:")
-        for task in xrange(len(self.tasks_name)):
-            logger.debug(3, " (%s)%s - %s: %s",
-                       task,
-                       self.fn_index[self.tasks_fnid[task]],
-                       self.tasks_name[task],
-                       self.tasks_tdepends[task])
+        bb.msg.debug(3, bb.msg.domain.TaskData, "tasks:")
+        for task in range(len(self.tasks_name)):
+            bb.msg.debug(3, bb.msg.domain.TaskData, " (%s)%s - %s: %s" % (
+                task,
+                self.fn_index[self.tasks_fnid[task]],
+                self.tasks_name[task],
+                self.tasks_tdepends[task]))
 
-        logger.debug(3, "dependency ids (per fn):")
+        bb.msg.debug(3, bb.msg.domain.TaskData, "dependency ids (per fn):")
         for fnid in self.depids:
-            logger.debug(3, " %s %s: %s", fnid, self.fn_index[fnid], self.depids[fnid])
+            bb.msg.debug(3, bb.msg.domain.TaskData, " %s %s: %s" % (fnid, self.fn_index[fnid], self.depids[fnid]))
 
-        logger.debug(3, "runtime dependency ids (per fn):")
+        bb.msg.debug(3, bb.msg.domain.TaskData, "runtime dependency ids (per fn):")
         for fnid in self.rdepids:
-            logger.debug(3, " %s %s: %s", fnid, self.fn_index[fnid], self.rdepids[fnid])
+            bb.msg.debug(3, bb.msg.domain.TaskData, " %s %s: %s" % (fnid, self.fn_index[fnid], self.rdepids[fnid]))

bitbake/lib/bb/ui/crumbs/__init__.py

@@ -1,5 +1,5 @@
 #
-# Gtk+ UI pieces for BitBake
+# BitBake UI Implementation
 #
 # Copyright (C) 2006-2007 Richard Purdie
 #

bitbake/lib/bb/ui/crumbs/configurator.py

@@ -1,346 +0,0 @@
-#
-# BitBake Graphical GTK User Interface
-#
-# Copyright (C) 2011        Intel Corporation
-#
-# Authored by Joshua Lock <josh@linux.intel.com>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-import gobject
-import copy
-import re, os
-from bb import data
-
-class Configurator(gobject.GObject):
-
-    """
-    A GObject to handle writing modified configuration values back
-    to conf files.
-    """
-    __gsignals__ = {
-        "layers-loaded"  : (gobject.SIGNAL_RUN_LAST,
-                           gobject.TYPE_NONE,
-                           ()),
-        "layers-changed" : (gobject.SIGNAL_RUN_LAST,
-                            gobject.TYPE_NONE,
-                            ())
-    }
-
-    def __init__(self):
-        gobject.GObject.__init__(self)
-        self.bblayers = None
-        self.enabled_layers = {}
-        self.loaded_layers = {}
-        self.config = {}
-        self.orig_config = {}
-        self.preconf = None
-        self.postconf = None
-
-    # NOTE: cribbed from the cooker...
-    def _parse(self, f, data, include=False):
-        try:
-            return bb.parse.handle(f, data, include)
-        except (IOError, bb.parse.ParseError) as exc:
-            parselog.critical("Unable to parse %s: %s" % (f, exc))
-            sys.exit(1)
-
-    def _loadConf(self, path):
-        def getString(var):
-            return bb.data.getVar(var, data, True) or ""
-
-        if self.orig_config:
-            del self.orig_config
-            self.orig_config = {}
-
-        data = bb.data.init()
-        data = self._parse(path, data)
-
-        # We only need to care about certain variables
-        mach = getString('MACHINE')
-        if mach and mach != self.config.get('MACHINE', ''):
-            self.config['MACHINE'] = mach
-        sdkmach = getString('SDKMACHINE')
-        if sdkmach and sdkmach != self.config.get('SDKMACHINE', ''):
-            self.config['SDKMACHINE'] = sdkmach
-        distro = getString('DISTRO')
-        if not distro:
-            distro = "defaultsetup"
-        if distro and distro != self.config.get('DISTRO', ''):
-            self.config['DISTRO'] = distro
-        bbnum = getString('BB_NUMBER_THREADS')
-        if bbnum and bbnum != self.config.get('BB_NUMBER_THREADS', ''):
-            self.config['BB_NUMBER_THREADS'] = bbnum
-        pmake = getString('PARALLEL_MAKE')
-        if pmake and pmake != self.config.get('PARALLEL_MAKE', ''):
-            self.config['PARALLEL_MAKE'] = pmake
-        pclass = getString('PACKAGE_CLASSES')
-        if pclass and pclass != self.config.get('PACKAGE_CLASSES', ''):
-            self.config['PACKAGE_CLASSES'] = pclass
-        fstypes = getString('IMAGE_FSTYPES')
-        if fstypes and fstypes != self.config.get('IMAGE_FSTYPES', ''):
-            self.config['IMAGE_FSTYPES'] = fstypes
-
-        # Values which aren't always set in the conf must be explicitly
-        # loaded as empty values for save to work
-        incompat = getString('INCOMPATIBLE_LICENSE')
-        if incompat and incompat != self.config.get('INCOMPATIBLE_LICENSE', ''):
-            self.config['INCOMPATIBLE_LICENSE'] = incompat
-        else:
-            self.config['INCOMPATIBLE_LICENSE'] = ""
-
-        # Non-standard, namespaces, variables for GUI preferences
-        toolchain = getString('HOB_BUILD_TOOLCHAIN')
-        if toolchain and toolchain != self.config.get('HOB_BUILD_TOOLCHAIN', ''):
-            self.config['HOB_BUILD_TOOLCHAIN'] = toolchain
-        header = getString('HOB_BUILD_TOOLCHAIN_HEADERS')
-        if header and header != self.config.get('HOB_BUILD_TOOLCHAIN_HEADERS', ''):
-            self.config['HOB_BUILD_TOOLCHAIN_HEADERS'] = header
-
-        self.orig_config = copy.deepcopy(self.config)
-
-    def setConfVar(self, var, val):
-        self.config[var] = val
-
-    def getConfVar(self, var):
-        if var in self.config:
-            return self.config[var]
-        else:
-            return ""
-
-    def _loadLayerConf(self, path):
-        self.bblayers = path
-        self.enabled_layers = {}
-        self.loaded_layers = {}
-        data = bb.data.init()
-        data = self._parse(self.bblayers, data)
-        layers = (bb.data.getVar('BBLAYERS', data, True) or "").split()
-        for layer in layers:
-            # TODO: we may be better off calling the layer by its
-            # BBFILE_COLLECTIONS value?
-            name = self._getLayerName(layer)
-            self.loaded_layers[name] = layer
-
-        self.enabled_layers = copy.deepcopy(self.loaded_layers)
-        self.emit("layers-loaded")
-
-    def _addConfigFile(self, path):
-        conffiles = ["local.conf", "hob-pre.conf", "hob-post.conf"]
-        pref, sep, filename = path.rpartition("/")
-
-        if filename == "hob-pre.conf":
-            self.preconf = path
-
-        if filename == "hob-post.conf":
-            self.postconf = path
-
-        if filename in conffiles:
-            self._loadConf(path)
-        elif filename == "bblayers.conf":
-            self._loadLayerConf(path)
-
-    def _splitLayer(self, path):
-        # we only care about the path up to /conf/layer.conf
-        layerpath, conf, end = path.rpartition("/conf/")
-        return layerpath
-
-    def _getLayerName(self, path):
-        # Should this be the collection name?
-        layerpath, sep, name = path.rpartition("/")
-        return name
-
-    def disableLayer(self, layer):
-        if layer in self.enabled_layers:
-            del self.enabled_layers[layer]
-
-    def addLayerConf(self, confpath):
-        layerpath = self._splitLayer(confpath)
-        name = self._getLayerName(layerpath)
-
-        if not layerpath or not name:
-            return None, None
-        elif name not in self.enabled_layers:
-            self.addLayer(name, layerpath)
-            return name, layerpath
-        else:
-            return name, None
-
-    def addLayer(self, name, path):
-        self.enabled_layers[name] = path
-
-    def _isLayerConfDirty(self):
-        # if a different number of layers enabled to what was
-        # loaded, definitely different
-        if len(self.enabled_layers) != len(self.loaded_layers):
-            return True
-
-        for layer in self.loaded_layers:
-            # if layer loaded but no longer present, definitely dirty
-            if layer not in self.enabled_layers:
-                return True
-
-        for layer in self.enabled_layers:
-            # if this layer wasn't present at load, definitely dirty
-            if layer not in self.loaded_layers:
-                return True
-            # if this layers path has changed, definitely dirty
-            if self.enabled_layers[layer] != self.loaded_layers[layer]:
-                return True
-
-        return False
-
-    def _constructLayerEntry(self):
-        """
-        Returns a string representing the new layer selection
-        """
-        layers = self.enabled_layers.copy()
-        # Construct BBLAYERS entry
-        layer_entry = "BBLAYERS = \" \\\n"
-        if 'meta' in layers:
-            layer_entry = layer_entry + "  %s \\\n" % layers['meta']
-            del layers['meta']
-        for layer in layers:
-            layer_entry = layer_entry + "  %s \\\n" % layers[layer]
-        layer_entry = layer_entry + "  \""
-
-        return "".join(layer_entry)
-
-    def writeConfFile(self, conffile, contents):
-        """
-        Make a backup copy of conffile and write a new file in its stead with
-        the lines in the contents list.
-        """
-        # Create a backup of the conf file
-        bkup = "%s~" % conffile
-        os.rename(conffile, bkup)
-
-        # Write the contents list object to the conf file
-        with open(conffile, "w") as new:
-            new.write("".join(contents))
-
-    def updateConf(self, orig_lines, changed_values):
-        new_config_lines = []
-        for var in changed_values:
-            # Convenience function for re.subn(). If the pattern matches
-            # return a string which contains an assignment using the same
-            # assignment operator as the old assignment.
-            def replace_val(matchobj):
-                var = matchobj.group(1) # config variable
-                op = matchobj.group(2) # assignment operator
-                val = changed_values[var] # new config value
-                return "%s %s \"%s\"" % (var, op, val)
-
-            pattern = '^\s*(%s)\s*([+=?.]+)(.*)' % re.escape(var)
-            p = re.compile(pattern)
-            cnt = 0
-            replaced = False
-
-            # Iterate over the local.conf lines and if they are a match
-            # for the pattern comment out the line and append a new line
-            # with the new VAR op "value" entry
-            for line in orig_lines:
-                new_line, replacements = p.subn(replace_val, line)
-                if replacements:
-                    orig_lines[cnt] = "#%s" % line
-                    new_config_lines.append(new_line)
-                    replaced = True
-                cnt = cnt + 1
-
-            if not replaced:
-                new_config_lines.append("%s = \"%s\"\n" % (var, changed_values[var]))
-
-        # Add the modified variables
-        orig_lines.extend(new_config_lines)
-        return orig_lines
-
-    def writeConf(self):
-        pre_vars = ["MACHINE", "SDKMACHINE", "DISTRO",
-                    "INCOMPATIBLE_LICENSE"]
-        post_vars = ["BB_NUMBER_THREADS", "PARALLEL_MAKE", "PACKAGE_CLASSES",
-                     "IMAGE_FSTYPES", "HOB_BUILD_TOOLCHAIN",
-                     "HOB_BUILD_TOOLCHAIN_HEADERS"]
-        pre_values = {}
-        post_values = {}
-        changed_values = {}
-        pre_lines = None
-        post_lines = None
-
-        for var in self.config:
-            val = self.config[var]
-            if self.orig_config.get(var, None) != val:
-                changed_values[var] = val
-
-        if not len(changed_values):
-            return
-
-        for var in changed_values:
-            if var in pre_vars:
-                pre_values[var] = changed_values[var]
-            elif var in post_vars:
-                post_values[var] = changed_values[var]
-
-        with open(self.preconf, 'r') as pre:
-            pre_lines = pre.readlines()
-        pre_lines = self.updateConf(pre_lines, pre_values)
-        if len(pre_lines):
-            self.writeConfFile(self.preconf, pre_lines)
-
-        with open(self.postconf, 'r') as post:
-            post_lines = post.readlines()
-        post_lines = self.updateConf(post_lines, post_values)
-        if len(post_lines):
-            self.writeConfFile(self.postconf, post_lines)
-
-        del self.orig_config
-        self.orig_config = copy.deepcopy(self.config)
-
-    def insertTempBBPath(self, bbpath, bbfiles):
-        # read the original conf into a list
-        with open(self.postconf, 'r') as config:
-            config_lines = config.readlines()
-
-        if bbpath:
-            config_lines.append("BBPATH := \"${BBPATH}:%s\"\n" % bbpath)
-        if bbfiles:
-            config_lines.append("BBFILES := \"${BBFILES} %s\"\n" % bbfiles)
-
-        self.writeConfFile(self.postconf, config_lines)
-
-    def writeLayerConf(self):
-        # If we've not added/removed new layers don't write
-        if not self._isLayerConfDirty():
-            return
-
-        # This pattern should find the existing BBLAYERS
-        pattern = 'BBLAYERS\s=\s\".*\"'
-
-        replacement = self._constructLayerEntry()
-
-        with open(self.bblayers, "r") as f:
-            contents = f.read()
-            p = re.compile(pattern, re.DOTALL)
-            new = p.sub(replacement, contents)
-
-        self.writeConfFile(self.bblayers, new)
-
-        # set loaded_layers for dirtiness tracking
-        self.loaded_layers = copy.deepcopy(self.enabled_layers)
-
-        self.emit("layers-changed")
-
-    def configFound(self, handler, path):
-        self._addConfigFile(path)
-
-    def loadConfig(self, path):
-        self._addConfigFile(path)

bitbake/lib/bb/ui/crumbs/hig.py

@@ -1,61 +0,0 @@
-#
-# BitBake Graphical GTK User Interface
-#
-# Copyright (C) 2011        Intel Corporation
-#
-# Authored by Joshua Lock <josh@linux.intel.com>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-import gobject
-import gtk
-"""
-The following are convenience classes for implementing GNOME HIG compliant
-BitBake GUI's
-In summary: spacing = 12px, border-width = 6px
-"""
-
-class CrumbsDialog(gtk.Dialog):
-    """
-    A GNOME HIG compliant dialog widget.
-    Add buttons with gtk.Dialog.add_button or gtk.Dialog.add_buttons
-    """
-    def __init__(self, parent=None, label="", icon=gtk.STOCK_INFO):
-        gtk.Dialog.__init__(self, "", parent, gtk.DIALOG_DESTROY_WITH_PARENT)
-        
-        #self.set_property("has-separator", False) # note: deprecated in 2.22
-
-        self.set_border_width(6)
-        self.vbox.set_property("spacing", 12)
-        self.action_area.set_property("spacing", 12)
-        self.action_area.set_property("border-width", 6)
-
-        first_row = gtk.HBox(spacing=12)
-        first_row.set_property("border-width", 6)
-        first_row.show()
-        self.vbox.add(first_row)
-
-        self.icon = gtk.Image()
-        self.icon.set_from_stock(icon, gtk.ICON_SIZE_DIALOG)
-        self.icon.set_property("yalign", 0.00)
-        self.icon.show()
-        first_row.add(self.icon)
-
-        self.label = gtk.Label()
-        self.label.set_use_markup(True)
-        self.label.set_line_wrap(True)
-        self.label.set_markup(label)
-        self.label.set_property("yalign", 0.00)
-        self.label.show()
-        first_row.add(self.label)

bitbake/lib/bb/ui/crumbs/hobeventhandler.py

@@ -1,343 +0,0 @@
-#
-# BitBake Graphical GTK User Interface
-#
-# Copyright (C) 2011        Intel Corporation
-#
-# Authored by Joshua Lock <josh@linux.intel.com>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-import gobject
-import logging
-import tempfile
-import datetime
-
-progress_total = 0
-
-class HobHandler(gobject.GObject):
-
-    """
-    This object does BitBake event handling for the hob gui.
-    """
-    __gsignals__ = {
-         "machines-updated"    : (gobject.SIGNAL_RUN_LAST,
-                                  gobject.TYPE_NONE,
-                                  (gobject.TYPE_PYOBJECT,)),
-         "sdk-machines-updated": (gobject.SIGNAL_RUN_LAST,
-                                  gobject.TYPE_NONE,
-                                  (gobject.TYPE_PYOBJECT,)),
-         "distros-updated"     : (gobject.SIGNAL_RUN_LAST,
-                                  gobject.TYPE_NONE,
-                                  (gobject.TYPE_PYOBJECT,)),
-         "package-formats-found" : (gobject.SIGNAL_RUN_LAST,
-                                  gobject.TYPE_NONE,
-                                  (gobject.TYPE_PYOBJECT,)),
-         "config-found"        : (gobject.SIGNAL_RUN_LAST,
-                                  gobject.TYPE_NONE,
-                                  (gobject.TYPE_STRING,)),
-         "generating-data"     : (gobject.SIGNAL_RUN_LAST,
-                                  gobject.TYPE_NONE,
-                                  ()),
-         "data-generated"      : (gobject.SIGNAL_RUN_LAST,
-                                  gobject.TYPE_NONE,
-                                  ()),
-         "fatal-error"         : (gobject.SIGNAL_RUN_LAST,
-                                  gobject.TYPE_NONE,
-                                  (gobject.TYPE_STRING,
-                                   gobject.TYPE_STRING,)),
-         "command-failed"      : (gobject.SIGNAL_RUN_LAST,
-                                  gobject.TYPE_NONE,
-                                  (gobject.TYPE_STRING,)),
-         "reload-triggered"    : (gobject.SIGNAL_RUN_LAST,
-                                  gobject.TYPE_NONE,
-                                  (gobject.TYPE_STRING,
-                                   gobject.TYPE_STRING,)),
-    }
-
-    (CFG_PATH_LOCAL, CFG_PATH_PRE, CFG_PATH_POST, CFG_PATH_LAYERS, CFG_FILES_DISTRO, CFG_FILES_MACH, CFG_FILES_SDK, FILES_MATCH_CLASS, GENERATE_TGTS, REPARSE_FILES, BUILD_IMAGE) = range(11)
-
-    def __init__(self, taskmodel, server):
-        gobject.GObject.__init__(self)
-
-        self.current_command = None
-        self.building = False
-        self.build_toolchain = False
-        self.build_toolchain_headers = False
-        self.generating = False
-        self.build_queue = []
-        self.current_phase = None
-        self.bbpath_ok = False
-        self.bbfiles_ok = False
-        self.build_type = "image"
-        self.image_dir = os.path.join(tempfile.gettempdir(), 'hob-images')
-
-        self.model = taskmodel
-        self.server = server
-
-        deploy_dir = self.server.runCommand(["getVariable", "DEPLOY_DIR"])
-        self.image_out_dir = os.path.join(deploy_dir, "images")
-        self.image_output_types = self.server.runCommand(["getVariable", "IMAGE_FSTYPES"]).split(" ")
-        self.bbpath = self.server.runCommand(["getVariable", "BBPATH"])
-        self.bbfiles = self.server.runCommand(["getVariable", "BBFILES"])
-
-    def run_next_command(self):
-        if self.current_command and not self.generating:
-            self.emit("generating-data")
-            self.generating = True
-
-        if self.current_command == self.CFG_PATH_LOCAL:
-            self.current_command = self.CFG_PATH_PRE
-            self.server.runCommand(["findConfigFilePath", "hob-pre.conf"])
-        elif self.current_command == self.CFG_PATH_PRE:
-            self.current_command = self.CFG_PATH_POST
-            self.server.runCommand(["findConfigFilePath", "hob-post.conf"])
-        elif self.current_command == self.CFG_PATH_POST:
-            self.current_command = self.CFG_PATH_LAYERS
-            self.server.runCommand(["findConfigFilePath", "bblayers.conf"])
-        elif self.current_command == self.CFG_PATH_LAYERS:
-            self.current_command = self.CFG_FILES_DISTRO
-            self.server.runCommand(["findConfigFiles", "DISTRO"])
-        elif self.current_command == self.CFG_FILES_DISTRO:
-            self.current_command = self.CFG_FILES_MACH
-            self.server.runCommand(["findConfigFiles", "MACHINE"])
-        elif self.current_command == self.CFG_FILES_MACH:
-            self.current_command = self.CFG_FILES_SDK
-            self.server.runCommand(["findConfigFiles", "MACHINE-SDK"])
-        elif self.current_command == self.CFG_FILES_SDK:
-            self.current_command = self.FILES_MATCH_CLASS
-            self.server.runCommand(["findFilesMatchingInDir", "rootfs_", "classes"])
-        elif self.current_command == self.FILES_MATCH_CLASS:
-            self.current_command = self.GENERATE_TGTS
-            self.server.runCommand(["generateTargetsTree", "classes/image.bbclass"])
-        elif self.current_command == self.GENERATE_TGTS:
-            if self.generating:
-                self.emit("data-generated")
-                self.generating = False
-            self.current_command = None
-        elif self.current_command == self.REPARSE_FILES:
-            if self.build_queue:
-                self.current_command = self.BUILD_IMAGE
-            else:
-                self.current_command = self.CFG_PATH_LAYERS
-            self.server.runCommand(["resetCooker"])
-            self.server.runCommand(["reparseFiles"])
-        elif self.current_command == self.BUILD_IMAGE:
-            if self.generating:
-                self.emit("data-generated")
-                self.generating = False
-            self.building = True
-            self.server.runCommand(["buildTargets", self.build_queue, "build"])
-            self.build_queue = []
-            self.current_command = None
-
-    def handle_event(self, event, running_build, pbar):
-        if not event:
-	    return
-
-        # If we're running a build, use the RunningBuild event handler
-        if self.building:
-            self.current_phase = "building"
-            running_build.handle_event(event)
-        elif isinstance(event, bb.event.TargetsTreeGenerated):
-            self.current_phase = "data generation"
-            if event._model:
-                self.model.populate(event._model)
-        elif isinstance(event, bb.event.ConfigFilesFound):
-            self.current_phase = "configuration lookup"
-            var = event._variable
-	    if var == "distro":
-		distros = event._values
-		distros.sort()
-		self.emit("distros-updated", distros)
-	    elif var == "machine":
-	        machines = event._values
-		machines.sort()
-		self.emit("machines-updated", machines)
-            elif var == "machine-sdk":
-                sdk_machines = event._values
-                sdk_machines.sort()
-                self.emit("sdk-machines-updated", sdk_machines)
-        elif isinstance(event, bb.event.ConfigFilePathFound):
-            self.current_phase = "configuration lookup"
-            path = event._path
-            self.emit("config-found", path)
-        elif isinstance(event, bb.event.FilesMatchingFound):
-            self.current_phase = "configuration lookup"
-            # FIXME: hard coding, should at least be a variable shared between
-            # here and the caller
-            if event._pattern == "rootfs_":
-                formats = []
-                for match in event._matches:
-                    classname, sep, cls = match.rpartition(".")
-                    fs, sep, format = classname.rpartition("_")
-                    formats.append(format)
-                formats.sort()
-                self.emit("package-formats-found", formats)
-        elif isinstance(event, bb.command.CommandCompleted):
-            self.current_phase = None
-            self.run_next_command()
-        elif isinstance(event, bb.command.CommandFailed):
-            self.emit("command-failed", event.error)
-        elif isinstance(event, bb.event.CacheLoadStarted):
-            self.current_phase = "cache loading"
-            bb.ui.crumbs.hobeventhandler.progress_total = event.total
-            pbar.set_text("Loading cache: %s/%s" % (0, bb.ui.crumbs.hobeventhandler.progress_total))
-        elif isinstance(event, bb.event.CacheLoadProgress):
-            self.current_phase = "cache loading"
-            pbar.set_text("Loading cache: %s/%s" % (event.current, bb.ui.crumbs.hobeventhandler.progress_total))
-        elif isinstance(event, bb.event.CacheLoadCompleted):
-            self.current_phase = None
-            pbar.set_text("Loading...")
-        elif isinstance(event, bb.event.ParseStarted):
-            self.current_phase = "recipe parsing"
-            if event.total == 0:
-                return
-            bb.ui.crumbs.hobeventhandler.progress_total = event.total
-            pbar.set_text("Processing recipes: %s/%s" % (0, bb.ui.crumbs.hobeventhandler.progress_total))
-        elif isinstance(event, bb.event.ParseProgress):
-            self.current_phase = "recipe parsing"
-            pbar.set_text("Processing recipes: %s/%s" % (event.current, bb.ui.crumbs.hobeventhandler.progress_total))
-        elif isinstance(event, bb.event.ParseCompleted):
-            self.current_phase = None
-            pbar.set_fraction(1.0)
-            pbar.set_text("Loading...")
-        elif isinstance(event, logging.LogRecord):
-            format = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
-            if event.levelno >= format.CRITICAL:
-                self.emit("fatal-error", event.getMessage(), self.current_phase)
-        return
-
-    def event_handle_idle_func (self, eventHandler, running_build, pbar):
-        # Consume as many messages as we can in the time available to us
-        event = eventHandler.getEvent()
-        while event:
-            self.handle_event(event, running_build, pbar)
-            event = eventHandler.getEvent()
-        return True
-
-    def set_machine(self, machine):
-        self.server.runCommand(["setVariable", "MACHINE", machine])
-
-    def set_sdk_machine(self, sdk_machine):
-        self.server.runCommand(["setVariable", "SDKMACHINE", sdk_machine])
-
-    def set_distro(self, distro):
-        self.server.runCommand(["setVariable", "DISTRO", distro])
-
-    def set_package_format(self, format):
-        self.server.runCommand(["setVariable", "PACKAGE_CLASSES", "package_%s" % format])
-
-    def reload_data(self, config=None):
-        img = self.model.selected_image
-        selected_packages, _ = self.model.get_selected_packages()
-        self.emit("reload-triggered", img, " ".join(selected_packages))
-        self.current_command = self.REPARSE_FILES
-        self.run_next_command()
-
-    def set_bbthreads(self, threads):
-        self.server.runCommand(["setVariable", "BB_NUMBER_THREADS", threads])
-
-    def set_pmake(self, threads):
-        pmake = "-j %s" % threads
-        self.server.runCommand(["setVariable", "BB_NUMBER_THREADS", pmake])
-
-    def build_targets(self, tgts, configurator, build_type="image"):
-        self.build_type = build_type
-        targets = []
-        nbbp = None
-        nbbf = None
-        targets.extend(tgts)
-        if self.build_toolchain and self.build_toolchain_headers:
-            targets.append("meta-toolchain-sdk")
-        elif self.build_toolchain:
-            targets.append("meta-toolchain")
-        self.build_queue = targets
-
-        if not self.bbpath_ok:
-            if self.image_dir in self.bbpath.split(":"):
-                self.bbpath_ok = True
-            else:
-                nbbp = self.image_dir
-
-        if not self.bbfiles_ok:
-            import re
-            pattern = "%s/\*.bb" % self.image_dir
-
-            for files in self.bbfiles.split(" "):
-                if re.match(pattern, files):
-                    self.bbfiles_ok = True
-
-            if not self.bbfiles_ok:
-                nbbf = "%s/*.bb" % self.image_dir
-
-        if nbbp or nbbf:
-            configurator.insertTempBBPath(nbbp, nbbf)
-            self.bbpath_ok = True
-            self.bbfiles_ok = True
-
-        self.current_command = self.REPARSE_FILES
-        self.run_next_command()
-
-    def cancel_build(self, force=False):
-        if force:
-            # Force the cooker to stop as quickly as possible
-            self.server.runCommand(["stateStop"])
-        else:
-            # Wait for tasks to complete before shutting down, this helps
-            # leave the workdir in a usable state
-            self.server.runCommand(["stateShutdown"])
-
-    def set_incompatible_license(self, incompatible):
-        self.server.runCommand(["setVariable", "INCOMPATIBLE_LICENSE", incompatible])
-
-    def toggle_toolchain(self, enabled):
-        if self.build_toolchain != enabled:
-            self.build_toolchain = enabled
-
-    def toggle_toolchain_headers(self, enabled):
-        if self.build_toolchain_headers != enabled:
-            self.build_toolchain_headers = enabled
-
-    def set_fstypes(self, fstypes):
-        self.server.runCommand(["setVariable", "IMAGE_FSTYPES", fstypes])
-
-    def add_image_output_type(self, output_type):
-        if output_type not in self.image_output_types:
-            self.image_output_types.append(output_type)
-            fstypes = " ".join(self.image_output_types).lstrip(" ")
-            self.set_fstypes(fstypes)
-        return self.image_output_types
-
-    def remove_image_output_type(self, output_type):
-        if output_type in self.image_output_types:
-            ind = self.image_output_types.index(output_type)
-            self.image_output_types.pop(ind)
-            fstypes = " ".join(self.image_output_types).lstrip(" ")
-            self.set_fstypes(fstypes)
-        return self.image_output_types
-
-    def get_image_deploy_dir(self):
-        return self.image_out_dir
-
-    def make_temp_dir(self):
-        bb.utils.mkdirhier(self.image_dir)
-
-    def remove_temp_dir(self):
-        bb.utils.remove(self.image_dir, True)
-
-    def get_temp_recipe_path(self, name):
-        timestamp = datetime.date.today().isoformat()
-        image_file = "hob-%s-variant-%s.bb" % (name, timestamp)
-        recipepath =  os.path.join(self.image_dir, image_file)
-        return recipepath

bitbake/lib/bb/ui/crumbs/hobprefs.py

@@ -1,335 +0,0 @@
-#
-# BitBake Graphical GTK User Interface
-#
-# Copyright (C) 2011        Intel Corporation
-#
-# Authored by Joshua Lock <josh@linux.intel.com>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-import gtk
-import glib
-from bb.ui.crumbs.configurator import Configurator
-
-class HobPrefs(gtk.Dialog):
-    """
-    """
-    def empty_combo_text(self, combo_text):
-        model = combo_text.get_model()
-        if model:
-            model.clear()
-
-    def output_type_toggled_cb(self, check, handler):
-        ot = check.get_label()
-        enabled = check.get_active()
-        if enabled:
-            self.selected_image_types = handler.add_image_output_type(ot)
-        else:
-            self.selected_image_types = handler.remove_image_output_type(ot)
-
-        self.configurator.setConfVar('IMAGE_FSTYPES', "%s" % " ".join(self.selected_image_types).lstrip(" "))
-        self.reload_required = True
-
-    def sdk_machine_combo_changed_cb(self, combo, handler):
-        sdk_mach = combo.get_active_text()
-	if sdk_mach != self.curr_sdk_mach:
-            self.curr_sdk_mach = sdk_mach
-            self.configurator.setConfVar('SDKMACHINE', sdk_mach)
-            handler.set_sdk_machine(sdk_mach)
-
-    def update_sdk_machines(self, handler, sdk_machines):
-        active = 0
-        # disconnect the signal handler before updating the combo model
-        if self.sdk_machine_handler_id:
-            self.sdk_machine_combo.disconnect(self.sdk_machine_handler_id)
-            self.sdk_machine_handler_id = None
-
-        self.empty_combo_text(self.sdk_machine_combo)
-        for sdk_machine in sdk_machines:
-            self.sdk_machine_combo.append_text(sdk_machine)
-            if sdk_machine == self.curr_sdk_mach:
-                self.sdk_machine_combo.set_active(active)
-            active = active + 1
-
-        self.sdk_machine_handler_id = self.sdk_machine_combo.connect("changed", self.sdk_machine_combo_changed_cb, handler)
-
-    def distro_combo_changed_cb(self, combo, handler):
-        distro = combo.get_active_text()
-	if distro != self.curr_distro:
-            self.curr_distro = distro
-            self.configurator.setConfVar('DISTRO', distro)
-            handler.set_distro(distro)
-            self.reload_required = True
-
-    def update_distros(self, handler, distros):
-        active = 0
-        # disconnect the signal handler before updating combo model
-        if self.distro_handler_id:
-            self.distro_combo.disconnect(self.distro_handler_id)
-            self.distro_handler_id = None
-
-        self.empty_combo_text(self.distro_combo)
-	for distro in distros:
-	    self.distro_combo.append_text(distro)
-	    if distro == self.curr_distro:
-                self.distro_combo.set_active(active)
-	    active = active + 1
-
-	self.distro_handler_id = self.distro_combo.connect("changed", self.distro_combo_changed_cb, handler)
-
-    def package_format_combo_changed_cb(self, combo, handler):
-        package_format = combo.get_active_text()
-        if package_format != self.curr_package_format:
-            self.curr_package_format = package_format
-            self.configurator.setConfVar('PACKAGE_CLASSES', 'package_%s' % package_format)
-            handler.set_package_format(package_format)
-            self.reload_required = True
-
-    def update_package_formats(self, handler, formats):
-        active = 0
-        # disconnect the signal handler before updating the model
-        if self.package_handler_id:
-            self.package_combo.disconnect(self.package_handler_id)
-            self.package_handler_id = None
-
-        self.empty_combo_text(self.package_combo)
-        for format in formats:
-            self.package_combo.append_text(format)
-            if format == self.curr_package_format:
-                self.package_combo.set_active(active)
-            active = active + 1
-
-        self.package_handler_id = self.package_combo.connect("changed", self.package_format_combo_changed_cb, handler)
-    
-    def include_gplv3_cb(self, toggle):
-        excluded = toggle.get_active()
-        orig_incompatible = self.configurator.getConfVar('INCOMPATIBLE_LICENSE')
-        new_incompatible = ""
-        if excluded:
-            if not orig_incompatible:
-                new_incompatible = "GPLv3"
-            elif not orig_incompatible.find('GPLv3'):
-                new_incompatible = "%s GPLv3" % orig_incompatible
-        else:
-            new_incompatible = orig_incompatible.replace('GPLv3', '')
-
-        if new_incompatible != orig_incompatible:
-            self.handler.set_incompatible_license(new_incompatible)
-            self.configurator.setConfVar('INCOMPATIBLE_LICENSE', new_incompatible)
-            self.reload_required = True
-
-    def change_bb_threads_cb(self, spinner):
-        val = spinner.get_value_as_int()
-        self.handler.set_bbthreads(val)
-        self.configurator.setConfVar('BB_NUMBER_THREADS', val)
-
-    def change_make_threads_cb(self, spinner):
-        val = spinner.get_value_as_int()
-        self.handler.set_pmake(val)
-        self.configurator.setConfVar('PARALLEL_MAKE', "-j %s" % val)
-
-    def toggle_toolchain_cb(self, check):
-        enabled = check.get_active()
-        toolchain = '0'
-        if enabled:
-            toolchain = '1'
-        self.handler.toggle_toolchain(enabled)
-        self.configurator.setConfVar('HOB_BUILD_TOOLCHAIN', toolchain)
-
-    def toggle_headers_cb(self, check):
-        enabled = check.get_active()
-        headers = '0'
-        if enabled:
-            headers = '1'
-        self.handler.toggle_toolchain_headers(enabled)
-        self.configurator.setConfVar('HOB_BUILD_TOOLCHAIN_HEADERS', headers)
-
-    def set_parent_window(self, parent):
-        self.set_transient_for(parent)
-
-    def write_changes(self):
-        self.configurator.writeConf()
-
-    def prefs_response_cb(self, dialog, response):
-        if self.reload_required:
-            glib.idle_add(self.handler.reload_data)
-            self.reload_required = False
-
-    def __init__(self, configurator, handler, curr_sdk_mach, curr_distro, pclass,
-                 pmake, bbthread, selected_image_types, all_image_types,
-                 gplv3disabled, build_toolchain, build_toolchain_headers):
-        """
-        """
-        gtk.Dialog.__init__(self, "Preferences", None,
-                            gtk.DIALOG_DESTROY_WITH_PARENT,
-                            (gtk.STOCK_CLOSE, gtk.RESPONSE_OK))
-
-        self.set_border_width(6)
-        self.vbox.set_property("spacing", 12)
-        self.action_area.set_property("spacing", 12)
-        self.action_area.set_property("border-width", 6)
-
-        self.handler = handler
-        self.configurator = configurator
-
-        self.curr_sdk_mach = curr_sdk_mach
-        self.curr_distro = curr_distro
-        self.curr_package_format = pclass
-        self.pmake = pmake
-        self.bbthread = bbthread
-        self.selected_image_types = selected_image_types.split(" ")
-        self.gplv3disabled = gplv3disabled
-        self.build_toolchain = build_toolchain
-        self.build_toolchain_headers = build_toolchain_headers
-
-        self.reload_required = False
-        self.distro_handler_id = None
-        self.sdk_machine_handler_id = None
-        self.package_handler_id = None
-
-        left = gtk.SizeGroup(gtk.SIZE_GROUP_HORIZONTAL)
-        right = gtk.SizeGroup(gtk.SIZE_GROUP_HORIZONTAL)
-
-        label = gtk.Label()
-        label.set_markup("<b>Policy</b>")
-        label.show()
-        frame = gtk.Frame()
-        frame.set_label_widget(label)
-        frame.set_shadow_type(gtk.SHADOW_NONE)
-        frame.show()
-        self.vbox.pack_start(frame)
-        pbox = gtk.VBox(False, 12)
-        pbox.show()
-        frame.add(pbox)
-        hbox = gtk.HBox(False, 12)
-        hbox.show()
-        pbox.pack_start(hbox, expand=False, fill=False, padding=6)
-        # Distro selector
-        label = gtk.Label("Distribution:")
-        label.show()
-        hbox.pack_start(label, expand=False, fill=False, padding=6)
-        self.distro_combo = gtk.combo_box_new_text()
-        self.distro_combo.set_tooltip_text("Select the Yocto distribution you would like to use")
-        self.distro_combo.show()
-        hbox.pack_start(self.distro_combo, expand=False, fill=False, padding=6)
-        # Exclude GPLv3
-        check = gtk.CheckButton("Exclude GPLv3 packages")
-        check.set_tooltip_text("Check this box to prevent GPLv3 packages from being included in your image")
-        check.show()
-        check.set_active(self.gplv3disabled)
-        check.connect("toggled", self.include_gplv3_cb)
-        hbox.pack_start(check, expand=False, fill=False, padding=6)
-        hbox = gtk.HBox(False, 12)
-        hbox.show()
-        pbox.pack_start(hbox, expand=False, fill=False, padding=6)
-        # Package format selector
-        label = gtk.Label("Package format:")
-        label.show()
-        hbox.pack_start(label, expand=False, fill=False, padding=6)
-        self.package_combo = gtk.combo_box_new_text()
-        self.package_combo.set_tooltip_text("""The package format is that used in creation
- of the root filesystem and also dictates the package manager used in your image""")
-        self.package_combo.show()
-        hbox.pack_start(self.package_combo, expand=False, fill=False, padding=6)
-        if all_image_types:
-            # Image output type selector
-            label = gtk.Label("Image output types:")
-            label.show()
-            hbox.pack_start(label, expand=False, fill=False, padding=6)
-            chk_cnt = 3
-            for it in all_image_types.split(" "):
-                chk_cnt = chk_cnt + 1
-                if chk_cnt % 6 == 0:
-                    hbox = gtk.HBox(False, 12)
-                    hbox.show()
-                    pbox.pack_start(hbox, expand=False, fill=False, padding=6)
-                chk = gtk.CheckButton(it)
-                if it in self.selected_image_types:
-                    chk.set_active(True)
-                chk.set_tooltip_text("Build an %s image" % it)
-                chk.connect("toggled", self.output_type_toggled_cb, handler)
-                chk.show()
-                hbox.pack_start(chk, expand=False, fill=False, padding=3)
-        # BitBake
-        label = gtk.Label()
-        label.set_markup("<b>BitBake</b>")
-        label.show()
-        frame = gtk.Frame()
-        frame.set_label_widget(label)
-        frame.set_shadow_type(gtk.SHADOW_NONE)
-        frame.show()
-        self.vbox.pack_start(frame)
-        pbox = gtk.VBox(False, 12)
-        pbox.show()
-        frame.add(pbox)
-        hbox = gtk.HBox(False, 12)
-        hbox.show()
-        pbox.pack_start(hbox, expand=False, fill=False, padding=6)
-        label = gtk.Label("BitBake threads:")
-        label.show()
-        # NOTE: may be a good idea in future to intelligently cap the maximum
-        # values but we need more data to make an educated decision, for now
-        # set a high maximum as a value for upper bounds is required by the
-        # gtk.Adjustment
-        spin_max = 30 # seems like a high enough arbitrary number
-        hbox.pack_start(label, expand=False, fill=False, padding=6)
-        bbadj = gtk.Adjustment(value=self.bbthread, lower=1, upper=spin_max, step_incr=1)
-        bbspinner = gtk.SpinButton(adjustment=bbadj, climb_rate=1, digits=0)
-        bbspinner.show()
-        bbspinner.connect("value-changed", self.change_bb_threads_cb)
-        hbox.pack_start(bbspinner, expand=False, fill=False, padding=6)
-        label = gtk.Label("Make threads:")
-        label.show()
-        hbox.pack_start(label, expand=False, fill=False, padding=6)
-        madj = gtk.Adjustment(value=self.pmake, lower=1, upper=spin_max, step_incr=1)
-        makespinner = gtk.SpinButton(adjustment=madj, climb_rate=1, digits=0)
-        makespinner.connect("value-changed", self.change_make_threads_cb)
-        makespinner.show()
-        hbox.pack_start(makespinner, expand=False, fill=False, padding=6)
-        # Toolchain
-        label = gtk.Label()
-        label.set_markup("<b>External Toolchain</b>")
-        label.show()
-        frame = gtk.Frame()
-        frame.set_label_widget(label)
-        frame.set_shadow_type(gtk.SHADOW_NONE)
-        frame.show()
-        self.vbox.pack_start(frame)
-        pbox = gtk.VBox(False, 12)
-        pbox.show()
-        frame.add(pbox)
-        hbox = gtk.HBox(False, 12)
-        hbox.show()
-        pbox.pack_start(hbox, expand=False, fill=False, padding=6)
-        toolcheck = gtk.CheckButton("Build external development toolchain with image")
-        toolcheck.show()
-        toolcheck.set_active(self.build_toolchain)
-        toolcheck.connect("toggled", self.toggle_toolchain_cb)
-        hbox.pack_start(toolcheck, expand=False, fill=False, padding=6)
-        hbox = gtk.HBox(False, 12)
-        hbox.show()
-        pbox.pack_start(hbox, expand=False, fill=False, padding=6)
-        label = gtk.Label("Toolchain host:")
-        label.show()
-        hbox.pack_start(label, expand=False, fill=False, padding=6)
-        self.sdk_machine_combo = gtk.combo_box_new_text()
-        self.sdk_machine_combo.set_tooltip_text("Select the host architecture of the external machine")
-        self.sdk_machine_combo.show()
-        hbox.pack_start(self.sdk_machine_combo, expand=False, fill=False, padding=6)
-        # headerscheck = gtk.CheckButton("Include development headers with toolchain")
-        # headerscheck.show()
-        # headerscheck.set_active(self.build_toolchain_headers)
-        # headerscheck.connect("toggled", self.toggle_headers_cb)
-        # hbox.pack_start(headerscheck, expand=False, fill=False, padding=6)
-        self.connect("response", self.prefs_response_cb)

bitbake/lib/bb/ui/crumbs/layereditor.py

@@ -1,153 +0,0 @@
-#
-# BitBake Graphical GTK User Interface
-#
-# Copyright (C) 2011        Intel Corporation
-#
-# Authored by Joshua Lock <josh@linux.intel.com>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-import gobject
-import gtk
-from bb.ui.crumbs.configurator import Configurator
-from bb.ui.crumbs.hig import CrumbsDialog
-
-class LayerEditor(gtk.Dialog):
-    """
-    Gtk+ Widget for enabling and disabling layers.
-    Layers are added through using an open dialog to find the layer.conf
-    Disabled layers are deleted from conf/bblayers.conf
-    """
-    def __init__(self, configurator, parent=None):
-        gtk.Dialog.__init__(self, "Layers", None,
-                            gtk.DIALOG_DESTROY_WITH_PARENT,
-                            (gtk.STOCK_CLOSE, gtk.RESPONSE_OK))
-
-        # We want to show a little more of the treeview in the default,
-        # emptier, case
-        self.set_size_request(-1, 300)
-        self.set_border_width(6)
-        self.vbox.set_property("spacing", 0)
-        self.action_area.set_property("border-width", 6)
-
-        self.configurator = configurator
-        self.newly_added = {}
-
-        # Label to inform users that meta is enabled but that you can't
-        # disable it as it'd be a *bad* idea
-        msg = "As the core of the build system the <i>meta</i> layer must always be included and therefore can't be viewed or edited here."
-        lbl = gtk.Label()
-        lbl.show()
-        lbl.set_use_markup(True)
-        lbl.set_markup(msg)
-        lbl.set_line_wrap(True)
-        lbl.set_justify(gtk.JUSTIFY_FILL)
-        self.vbox.pack_start(lbl, expand=False, fill=False, padding=6)
-
-        # Create a treeview in which to list layers
-        # ListStore of Name, Path, Enabled
-        self.layer_store = gtk.ListStore(gobject.TYPE_STRING, gobject.TYPE_STRING, gobject.TYPE_BOOLEAN)
-        self.tv = gtk.TreeView(self.layer_store)
-        self.tv.set_headers_visible(True)
-
-        col0 = gtk.TreeViewColumn('Name')
-        self.tv.append_column(col0)
-        col1 = gtk.TreeViewColumn('Path')
-        self.tv.append_column(col1)
-        col2 = gtk.TreeViewColumn('Enabled')
-        self.tv.append_column(col2)
-
-        cell0 = gtk.CellRendererText()
-        col0.pack_start(cell0, True)
-        col0.set_attributes(cell0, text=0)
-        cell1 = gtk.CellRendererText()
-        col1.pack_start(cell1, True)
-        col1.set_attributes(cell1, text=1)
-        cell2 = gtk.CellRendererToggle()
-        cell2.connect("toggled", self._toggle_layer_cb)
-        col2.pack_start(cell2, True)
-        col2.set_attributes(cell2, active=2)
-
-        self.tv.show()
-        self.vbox.pack_start(self.tv, expand=True, fill=True, padding=0)
-
-        tb = gtk.Toolbar()
-        tb.set_icon_size(gtk.ICON_SIZE_SMALL_TOOLBAR)
-        tb.set_style(gtk.TOOLBAR_BOTH)
-        tb.set_tooltips(True)
-        tb.show()
-        icon = gtk.Image()
-        icon.set_from_stock(gtk.STOCK_ADD, gtk.ICON_SIZE_SMALL_TOOLBAR)
-        icon.show()
-        tb.insert_item("Add Layer", "Add new layer", None, icon,
-                       self._find_layer_cb, None, -1)
-        self.vbox.pack_start(tb, expand=False, fill=False, padding=0)
-
-    def set_parent_window(self, parent):
-        self.set_transient_for(parent)
-
-    def load_current_layers(self, data):
-        for layer, path in self.configurator.enabled_layers.items():
-            if layer != 'meta':
-                self.layer_store.append([layer, path, True])
-
-    def save_current_layers(self):
-        self.configurator.writeLayerConf()
-
-    def _toggle_layer_cb(self, cell, path):
-        name = self.layer_store[path][0]
-        toggle = not self.layer_store[path][2]
-        if toggle:
-            self.configurator.addLayer(name, path)
-        else:
-            self.configurator.disableLayer(name)
-        self.layer_store[path][2] = toggle
-
-    def _find_layer_cb(self, button):
-        self.find_layer(self)
-
-    def find_layer(self, parent):
-        def conf_error(parent, lbl):
-            dialog = CrumbsDialog(parent, lbl)
-            dialog.add_button(gtk.STOCK_OK, gtk.RESPONSE_OK)
-            response = dialog.run()
-            dialog.destroy()
-
-        dialog = gtk.FileChooserDialog("Add new layer", parent,
-                                       gtk.FILE_CHOOSER_ACTION_OPEN,
-                                       (gtk.STOCK_CANCEL, gtk.RESPONSE_NO,
-                                        gtk.STOCK_OPEN, gtk.RESPONSE_YES))
-        label = gtk.Label("Select the layer.conf of the layer you wish to add")
-        label.show()
-        dialog.set_extra_widget(label)
-        response = dialog.run()
-        path = dialog.get_filename()
-        dialog.destroy()
-
-        lbl = "<b>Error</b>\nUnable to load layer <i>%s</i> because " % path
-        if response == gtk.RESPONSE_YES:
-            # FIXME: verify we've actually got a layer conf?
-            if path.endswith("layer.conf"):
-                name, layerpath = self.configurator.addLayerConf(path)
-                if name and layerpath:
-                    self.newly_added[name] = layerpath
-                    self.layer_store.append([name, layerpath, True])
-                    return
-                elif name:
-                    return
-                else:
-                    lbl += "there was a problem parsing the layer.conf."
-            else:
-                lbl += "it is not a layer.conf file."
-            conf_error(parent, lbl)

bitbake/lib/bb/ui/crumbs/progress.py

@@ -1,20 +0,0 @@
-import gtk
-
-class ProgressBar(gtk.Dialog):
-    def __init__(self, parent):
-
-        gtk.Dialog.__init__(self, flags=(gtk.DIALOG_MODAL | gtk.DIALOG_DESTROY_WITH_PARENT))
-        self.set_title("Parsing metadata, please wait...")
-        self.set_default_size(500, 0)
-        self.set_transient_for(parent)
-        self.progress = gtk.ProgressBar()
-        self.vbox.pack_start(self.progress)
-        self.show_all()
-
-    def update(self, x, y):
-        self.progress.set_fraction(float(x)/float(y))
-        self.progress.set_text("%2d %%" % (x*100/y))
-
-    def pulse(self):
-        self.progress.set_text("Loading...")
-        self.progress.pulse()

bitbake/lib/bb/ui/crumbs/runningbuild.py

@@ -1,4 +1,3 @@
-
 #
 # BitBake Graphical GTK User Interface
 #
@@ -21,20 +20,9 @@
 
 import gtk
 import gobject
-import logging
-import time
-import urllib
-import urllib2
-
-class Colors(object):
-    OK = "#ffffff"
-    RUNNING = "#aaffaa"
-    WARNING ="#f88017"
-    ERROR = "#ffaaaa"
 
 class RunningBuildModel (gtk.TreeStore):
-    (COL_LOG, COL_PACKAGE, COL_TASK, COL_MESSAGE, COL_ICON, COL_COLOR, COL_NUM_ACTIVE) = range(7)
-
+    (COL_TYPE, COL_PACKAGE, COL_TASK, COL_MESSAGE, COL_ICON, COL_ACTIVE) = (0, 1, 2, 3, 4, 5)
     def __init__ (self):
         gtk.TreeStore.__init__ (self,
                                 gobject.TYPE_STRING,
@@ -42,38 +30,25 @@ class RunningBuildModel (gtk.TreeStore):
                                 gobject.TYPE_STRING,
                                 gobject.TYPE_STRING,
                                 gobject.TYPE_STRING,
-                                gobject.TYPE_STRING,
-                                gobject.TYPE_INT)
+                                gobject.TYPE_BOOLEAN)
 
 class RunningBuild (gobject.GObject):
     __gsignals__ = {
-          'build-started' : (gobject.SIGNAL_RUN_LAST,
-                               gobject.TYPE_NONE,
-                               ()),
           'build-succeeded' : (gobject.SIGNAL_RUN_LAST,
                                gobject.TYPE_NONE,
                                ()),
           'build-failed' : (gobject.SIGNAL_RUN_LAST,
                             gobject.TYPE_NONE,
-                            ()),
-          'build-complete' : (gobject.SIGNAL_RUN_LAST,
-                              gobject.TYPE_NONE,
-                              ())
+                            ())
           }
     pids_to_task = {}
     tasks_to_iter = {}
 
-    def __init__ (self, sequential=False):
+    def __init__ (self):
         gobject.GObject.__init__ (self)
         self.model = RunningBuildModel()
-        self.sequential = sequential
 
-    def reset (self):
-        self.pids_to_task.clear()
-        self.tasks_to_iter.clear()
-        self.model.clear()
-
-    def handle_event (self, event, pbar=None):
+    def handle_event (self, event):
         # Handle an event from the event queue, this may result in updating
         # the model and thus the UI. Or it may be to tell us that the build
         # has finished successfully (or not, as the case may be.)
@@ -88,49 +63,32 @@ class RunningBuild (gobject.GObject):
         # for the message.
         if hasattr(event, 'pid'):
             pid = event.pid
-        if hasattr(event, 'process'):
-            pid = event.process
+            if pid in self.pids_to_task:
+                (package, task) = self.pids_to_task[pid]
+                parent = self.tasks_to_iter[(package, task)]
 
-        if pid and pid in self.pids_to_task:
-            (package, task) = self.pids_to_task[pid]
-            parent = self.tasks_to_iter[(package, task)]
-
-        if(isinstance(event, logging.LogRecord)):
-            # FIXME: this is a hack! More info in Yocto #1433
-            # http://bugzilla.pokylinux.org/show_bug.cgi?id=1433, temporarily
-            # mask the error message as it's not informative for the user.
-            if event.msg.startswith("Execution of event handler 'run_buildstats' failed"):
-                return
-
-            if (event.levelno < logging.INFO or
-                event.msg.startswith("Running task")):
-                return # don't add these to the list
-
-            if event.levelno >= logging.ERROR:
-                icon = "dialog-error"
-                color = Colors.ERROR
-            elif event.levelno >= logging.WARNING:
+        if isinstance(event, bb.msg.Msg):
+            # Set a pretty icon for the message based on it's type.
+            if isinstance(event, bb.msg.MsgWarn):
                 icon = "dialog-warning"
-                color = Colors.WARNING
+            elif isinstance(event, bb.msg.MsgErr):
+                icon = "dialog-error"
             else:
                 icon = None
-                color = Colors.OK
 
-            # if we know which package we belong to, we'll append onto its list.
-            # otherwise, we'll jump to the top of the master list
-            if self.sequential or not parent:
-                tree_add = self.model.append
-            else:
-                tree_add = self.model.prepend
-            tree_add(parent,
-                     (None,
-                      package,
-                      task,
-                      event.getMessage(),
-                      icon,
-                      color,
-                      0))
+            # Ignore the "Running task i of n .." messages
+            if (event._message.startswith ("Running task")):
+                return
 
+            # Add the message to the tree either at the top level if parent is
+            # None otherwise as a descendent of a task.
+            self.model.append (parent,
+                               (event.__name__.split()[-1], # e.g. MsgWarn, MsgError
+                                package,
+                                task,
+                                event._message,
+                                icon,
+                                False))
         elif isinstance(event, bb.build.TaskStarted):
             (package, task) = (event._package, event._task)
 
@@ -143,165 +101,70 @@ class RunningBuild (gobject.GObject):
             if ((package, None) in self.tasks_to_iter):
                 parent = self.tasks_to_iter[(package, None)]
             else:
-                if self.sequential:
-                    add = self.model.append
-                else:
-                    add = self.model.prepend
-                parent = add(None, (None,
-                                    package,
-                                    None,
-                                    "Package: %s" % (package),
-                                    None,
-                                    Colors.OK,
-                                    0))
+                parent = self.model.append (None, (None,
+                                                   package,
+                                                   None,
+                                                   "Package: %s" % (package),
+                                                   None,
+                                                   False))
                 self.tasks_to_iter[(package, None)] = parent
 
             # Because this parent package now has an active child mark it as
             # such.
-            # @todo if parent is already in error, don't mark it green
-            self.model.set(parent, self.model.COL_ICON, "gtk-execute",
-                           self.model.COL_COLOR, Colors.RUNNING)
+            self.model.set(parent, self.model.COL_ICON, "gtk-execute")
 
             # Add an entry in the model for this task
             i = self.model.append (parent, (None,
                                             package,
                                             task,
                                             "Task: %s" % (task),
-                                            "gtk-execute",
-                                            Colors.RUNNING,
-                                            0))
-
-            # update the parent's active task count
-            num_active = self.model.get(parent, self.model.COL_NUM_ACTIVE)[0] + 1
-            self.model.set(parent, self.model.COL_NUM_ACTIVE, num_active)
+                                            None,
+                                            False))
 
             # Save out the iter so that we can find it when we have a message
             # that we need to attach to a task.
             self.tasks_to_iter[(package, task)] = i
 
-        # If we don't handle these the GUI does not proceed
-        elif isinstance(event, bb.build.TaskInvalid):
-            return
+            # Mark this task as active.
+            self.model.set(i, self.model.COL_ICON, "gtk-execute")
 
-        elif isinstance(event, bb.build.TaskBase):
-            current = self.tasks_to_iter[(package, task)]
-            parent = self.tasks_to_iter[(package, None)]
-
-            # remove this task from the parent's active count
-            num_active = self.model.get(parent, self.model.COL_NUM_ACTIVE)[0] - 1
-            self.model.set(parent, self.model.COL_NUM_ACTIVE, num_active)
+        elif isinstance(event, bb.build.Task):
 
             if isinstance(event, bb.build.TaskFailed):
-                # Mark the task and parent as failed
-                icon = "dialog-error"
-                color = Colors.ERROR
+                # Mark the task as failed
+                i = self.tasks_to_iter[(package, task)]
+                self.model.set(i, self.model.COL_ICON, "dialog-error")
 
-                logfile = event.logfile
-                if logfile and os.path.exists(logfile):
-                    with open(logfile) as f:
-                        logdata = f.read()
-                        self.model.append(current, ('pastebin', None, None, logdata, 'gtk-error', Colors.OK, 0))
-
-                for i in (current, parent):
-                    self.model.set(i, self.model.COL_ICON, icon,
-                                   self.model.COL_COLOR, color)
-            else:
-                icon = None
-                color = Colors.OK
-
-                # Mark the task as inactive
-                self.model.set(current, self.model.COL_ICON, icon,
-                               self.model.COL_COLOR, color)
-
-                # Mark the parent package as inactive, but make sure to
-                # preserve error and active states
+                # Mark the parent package as failed
                 i = self.tasks_to_iter[(package, None)]
-                if self.model.get(parent, self.model.COL_ICON) != 'dialog-error':
-                    self.model.set(parent, self.model.COL_ICON, icon)
-                    if num_active == 0:
-                        self.model.set(parent, self.model.COL_COLOR, Colors.OK)
+                self.model.set(i, self.model.COL_ICON, "dialog-error")
+            else:
+                # Mark the task as inactive
+                i = self.tasks_to_iter[(package, task)]
+                self.model.set(i, self.model.COL_ICON, None)
+
+                # Mark the parent package as inactive
+                i = self.tasks_to_iter[(package, None)]
+                self.model.set(i, self.model.COL_ICON, None)
+
 
             # Clear the iters and the pids since when the task goes away the
             # pid will no longer be used for messages
             del self.tasks_to_iter[(package, task)]
             del self.pids_to_task[pid]
 
-        elif isinstance(event, bb.event.BuildStarted):
-
-            self.emit("build-started")
-            self.model.prepend(None, (None,
-                                      None,
-                                      None,
-                                      "Build Started (%s)" % time.strftime('%m/%d/%Y %H:%M:%S'),
-                                      None,
-                                      Colors.OK,
-                                      0))
         elif isinstance(event, bb.event.BuildCompleted):
             failures = int (event._failures)
-            self.model.prepend(None, (None,
-                                      None,
-                                      None,
-                                      "Build Completed (%s)" % time.strftime('%m/%d/%Y %H:%M:%S'),
-                                      None,
-                                      Colors.OK,
-                                      0))
 
             # Emit the appropriate signal depending on the number of failures
-            if (failures >= 1):
+            if (failures > 1):
                 self.emit ("build-failed")
             else:
                 self.emit ("build-succeeded")
-            # Emit a generic "build-complete" signal for things wishing to
-            # handle when the build is finished
-            self.emit("build-complete")
-
-        elif isinstance(event, bb.command.CommandFailed):
-            if event.error.startswith("Exited with"):
-                # If the command fails with an exit code we're done, emit the
-                # generic signal for the UI to notify the user
-                self.emit("build-complete")
-
-        elif isinstance(event, bb.event.CacheLoadStarted) and pbar:
-            pbar.set_title("Loading cache")
-            self.progress_total = event.total
-            pbar.update(0, self.progress_total)
-        elif isinstance(event, bb.event.CacheLoadProgress) and pbar:
-            pbar.update(event.current, self.progress_total)
-        elif isinstance(event, bb.event.CacheLoadCompleted) and pbar:
-            pbar.update(self.progress_total, self.progress_total)
-            pbar.hide()
-        elif isinstance(event, bb.event.ParseStarted) and pbar:
-            if event.total == 0:
-                return
-            pbar.set_title("Processing recipes")
-            self.progress_total = event.total
-            pbar.update(0, self.progress_total)
-        elif isinstance(event, bb.event.ParseProgress) and pbar:
-            pbar.update(event.current, self.progress_total)
-        elif isinstance(event, bb.event.ParseCompleted) and pbar:
-            pbar.hide()
-
-        return
-
-
-def do_pastebin(text):
-    url = 'http://pastebin.com/api_public.php'
-    params = {'paste_code': text, 'paste_format': 'text'}
-
-    req = urllib2.Request(url, urllib.urlencode(params))
-    response = urllib2.urlopen(req)
-    paste_url = response.read()
-
-    return paste_url
-
 
 class RunningBuildTreeView (gtk.TreeView):
-    __gsignals__ = {
-        "button_press_event" : "override"
-        }
-    def __init__ (self, readonly=False):
+    def __init__ (self):
         gtk.TreeView.__init__ (self)
-        self.readonly = readonly
 
         # The icon that indicates whether we're building or failed.
         renderer = gtk.CellRendererPixbuf ()
@@ -310,60 +173,6 @@ class RunningBuildTreeView (gtk.TreeView):
         self.append_column (col)
 
         # The message of the build.
-        self.message_renderer = gtk.CellRendererText ()
-        self.message_column = gtk.TreeViewColumn ("Message", self.message_renderer, text=3)
-        self.message_column.add_attribute(self.message_renderer, 'background', 5)
-        self.message_renderer.set_property('editable', (not self.readonly))
-        self.append_column (self.message_column)
-
-    def do_button_press_event(self, event):
-        gtk.TreeView.do_button_press_event(self, event)
-
-        if event.button == 3:
-            selection = super(RunningBuildTreeView, self).get_selection()
-            (model, it) = selection.get_selected()
-            if it is not None:
-                can_paste = model.get(it, model.COL_LOG)[0]
-                if can_paste == 'pastebin':
-                    # build a simple menu with a pastebin option
-                    menu = gtk.Menu()
-                    menuitem = gtk.MenuItem("Copy")
-                    menu.append(menuitem)
-                    menuitem.connect("activate", self.clipboard_handler, (model, it))
-                    menuitem.show()
-                    menuitem = gtk.MenuItem("Send log to pastebin")
-                    menu.append(menuitem)
-                    menuitem.connect("activate", self.pastebin_handler, (model, it))
-                    menuitem.show()
-                    menu.show()
-                    menu.popup(None, None, None, event.button, event.time)
-
-    def _add_to_clipboard(self, clipping):
-        """
-        Add the contents of clipping to the system clipboard.
-        """
-        clipboard = gtk.clipboard_get()
-        clipboard.set_text(clipping)
-        clipboard.store()
-
-    def pastebin_handler(self, widget, data):
-        """
-        Send the log data to pastebin, then add the new paste url to the
-        clipboard.
-        """
-        (model, it) = data
-        paste_url = do_pastebin(model.get(it, model.COL_MESSAGE)[0])
-
-        # @todo Provide visual feedback to the user that it is done and that
-        # it worked.
-        print paste_url
-
-        self._add_to_clipboard(paste_url)
-
-    def clipboard_handler(self, widget, data):
-        """
-        """
-        (model, it) = data
-        message = model.get(it, model.COL_MESSAGE)[0]
-
-        self._add_to_clipboard(message)
+        renderer = gtk.CellRendererText ()
+        col = gtk.TreeViewColumn ("Message", renderer, text=3)
+        self.append_column (col)

bitbake/lib/bb/ui/crumbs/tasklistmodel.py

@@ -1,620 +0,0 @@
-#
-# BitBake Graphical GTK User Interface
-#
-# Copyright (C) 2011        Intel Corporation
-#
-# Authored by Joshua Lock <josh@linux.intel.com>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-import gtk
-import gobject
-import re
-
-class BuildRep(gobject.GObject):
-
-    def __init__(self, userpkgs, allpkgs, base_image=None):
-        gobject.GObject.__init__(self)
-        self.base_image = base_image
-        self.allpkgs = allpkgs
-        self.userpkgs = userpkgs
-
-    def loadRecipe(self, pathname):
-        contents = []
-        packages = ""
-        base_image = ""
-
-        with open(pathname, 'r') as f:
-            contents = f.readlines()
-
-        pkg_pattern = "^\s*(IMAGE_INSTALL)\s*([+=.?]+)\s*(\".*?\")"
-        img_pattern = "^\s*(require)\s+(\S+.bb)"
-
-        for line in contents:
-            matchpkg = re.search(pkg_pattern, line)
-            matchimg = re.search(img_pattern, line)
-            if matchpkg:
-                packages = packages + matchpkg.group(3).strip('"')
-            if matchimg:
-                base_image = os.path.basename(matchimg.group(2)).split(".")[0]
-
-        self.base_image = base_image
-        self.userpkgs = packages
-
-    def writeRecipe(self, writepath, model):
-        template = """
-# Recipe generated by the HOB
-
-require %s
-
-IMAGE_INSTALL += "%s"
-"""
-
-        empty_template = """
-# Recipe generated by the HOB
-
-inherit core-image
-
-IMAGE_INSTALL = "%s"
-"""
-        if self.base_image and not self.base_image == "empty":
-            meta_path = model.find_image_path(self.base_image)
-            recipe = template % (meta_path, self.userpkgs)
-        else:
-            recipe = empty_template % self.allpkgs
-
-        if os.path.exists(writepath):
-            os.rename(writepath, "%s~" % writepath)
-
-        with open(writepath, 'w') as r:
-            r.write(recipe)
-
-        return writepath
-
-class TaskListModel(gtk.ListStore):
-    """
-    This class defines an gtk.ListStore subclass which will convert the output
-    of the bb.event.TargetsTreeGenerated event into a gtk.ListStore whilst also
-    providing convenience functions to access gtk.TreeModel subclasses which
-    provide filtered views of the data.
-    """
-    (COL_NAME, COL_DESC, COL_LIC, COL_GROUP, COL_DEPS, COL_BINB, COL_TYPE, COL_INC, COL_IMG, COL_PATH, COL_PN) = range(11)
-
-    __gsignals__ = {
-        "tasklist-populated" : (gobject.SIGNAL_RUN_LAST,
-                                gobject.TYPE_NONE,
-                                ()),
-        "contents-changed"   : (gobject.SIGNAL_RUN_LAST,
-                                gobject.TYPE_NONE,
-                                (gobject.TYPE_INT,)),
-        "image-changed"      : (gobject.SIGNAL_RUN_LAST,
-                                gobject.TYPE_NONE,
-                                (gobject.TYPE_STRING,)),
-        }
-
-    """
-    """
-    def __init__(self):
-        self.contents = None
-        self.tasks = None
-        self.packages = None
-        self.images = None
-        self.selected_image = None
-        
-        gtk.ListStore.__init__ (self,
-                                gobject.TYPE_STRING,
-                                gobject.TYPE_STRING,
-                                gobject.TYPE_STRING,
-                                gobject.TYPE_STRING,
-                                gobject.TYPE_STRING,
-                                gobject.TYPE_STRING,
-                                gobject.TYPE_STRING,
-                                gobject.TYPE_BOOLEAN,
-                                gobject.TYPE_BOOLEAN,
-                                gobject.TYPE_STRING,
-                                gobject.TYPE_STRING)
-
-    """
-    Helper method to determine whether name is a target pn
-    """
-    def non_target_name(self, name):
-        if ('-native' in name) or ('-cross' in name) or name.startswith('virtual/'):
-            return True
-        return False
-
-    def contents_changed_cb(self, tree_model, path, it=None):
-        pkg_cnt = self.contents.iter_n_children(None)
-        self.emit("contents-changed", pkg_cnt)
-
-    def contents_model_filter(self, model, it):
-        if not model.get_value(it, self.COL_INC) or model.get_value(it, self.COL_TYPE) == 'image':
-            return False
-        name = model.get_value(it, self.COL_NAME)
-        if self.non_target_name(name):
-            return False
-        else:
-            return True
-
-    """
-    Create, if required, and return a filtered gtk.TreeModel
-    containing only the items which are to be included in the
-    image
-    """
-    def contents_model(self):
-        if not self.contents:
-            self.contents = self.filter_new()
-            self.contents.set_visible_func(self.contents_model_filter)
-            self.contents.connect("row-inserted", self.contents_changed_cb)
-            self.contents.connect("row-deleted", self.contents_changed_cb)
-        return self.contents
-    
-    """
-    Helper function to determine whether an item is a task
-    """
-    def task_model_filter(self, model, it):
-        if model.get_value(it, self.COL_TYPE) == 'task':
-            return True
-        else:
-            return False
-
-    """
-    Create, if required, and return a filtered gtk.TreeModel
-    containing only the items which are tasks
-    """
-    def tasks_model(self):
-        if not self.tasks:
-            self.tasks = self.filter_new()
-            self.tasks.set_visible_func(self.task_model_filter)
-        return self.tasks
-
-    """
-    Helper function to determine whether an item is an image
-    """
-    def image_model_filter(self, model, it):
-        if model.get_value(it, self.COL_TYPE) == 'image':
-            return True
-        else:
-            return False
-
-    """
-    Create, if required, and return a filtered gtk.TreeModel
-    containing only the items which are images
-    """
-    def images_model(self):
-        if not self.images:
-            self.images = self.filter_new()
-            self.images.set_visible_func(self.image_model_filter)
-        return self.images
-
-    """
-    Helper function to determine whether an item is a package
-    """
-    def package_model_filter(self, model, it):
-        if model.get_value(it, self.COL_TYPE) != 'package':
-            return False
-        else:
-            name = model.get_value(it, self.COL_NAME)
-            if self.non_target_name(name):
-                return False
-            return True
-
-    """
-    Create, if required, and return a filtered gtk.TreeModel
-    containing only the items which are packages
-    """
-    def packages_model(self):
-        if not self.packages:
-            self.packages = self.filter_new()
-            self.packages.set_visible_func(self.package_model_filter)
-        return self.packages
-
-    """
-    The populate() function takes as input the data from a
-    bb.event.TargetsTreeGenerated event and populates the TaskList.
-    Once the population is done it emits gsignal tasklist-populated
-    to notify any listeners that the model is ready
-    """
-    def populate(self, event_model):
-        # First clear the model, in case repopulating
-        self.clear()
-        for item in event_model["pn"]:
-            atype = 'package'
-            name = item
-            summary = event_model["pn"][item]["summary"]
-            lic = event_model["pn"][item]["license"]
-            group = event_model["pn"][item]["section"]
-            filename = event_model["pn"][item]["filename"]
-            if ('task-' in name):
-                atype = 'task'
-            elif ('-image-' in name):
-                atype = 'image'
-
-            # Create a combined list of build and runtime dependencies and
-            # then remove any duplicate entries and any entries for -dev
-            # packages
-            depends = event_model["depends"].get(item, [])
-            rdepends = event_model["rdepends-pn"].get(item, [])
-            packages = {}
-            for pkg in event_model["packages"]:
-                if event_model["packages"][pkg]["pn"] == name:
-                    deps = []
-                    deps.extend(depends)
-                    deps.extend(event_model["rdepends-pkg"].get(pkg, []))
-                    deps.extend(rdepends)
-                    deps = self.squish(deps)
-                    # rdepends-pn includes pn-dev
-                    if ("%s-dev" % item) in deps:
-                        deps.remove("%s-dev" % item)
-                    # rdepends-on includes pn
-                    if pkg in deps:
-                        deps.remove(pkg)
-                    packages[pkg] = deps
-
-            for p in packages:
-                self.set(self.append(), self.COL_NAME, p, self.COL_DESC, summary,
-                         self.COL_LIC, lic, self.COL_GROUP, group,
-                         self.COL_DEPS, " ".join(packages[p]), self.COL_BINB, "",
-                         self.COL_TYPE, atype, self.COL_INC, False,
-                         self.COL_IMG, False, self.COL_PATH, filename,
-                         self.COL_PN, item)
-
-	self.emit("tasklist-populated")
-
-    """
-    Load a BuildRep into the model
-    """
-    def load_image_rep(self, rep):
-        # Unset everything
-        it = self.get_iter_first()
-        while it:
-            path = self.get_path(it)
-            self[path][self.COL_INC] = False
-            self[path][self.COL_IMG] = False
-            it = self.iter_next(it)
-
-        # Iterate the images and disable them all
-        it = self.images.get_iter_first()
-        while it:
-            path = self.images.convert_path_to_child_path(self.images.get_path(it))
-            name = self[path][self.COL_NAME]
-            if name == rep.base_image:
-                self.include_item(path, image_contents=True)
-            else:
-                self[path][self.COL_INC] = False
-            it = self.images.iter_next(it)
-
-        # Mark all of the additional packages for inclusion
-        packages = rep.userpkgs.split(" ")
-        it = self.get_iter_first()
-        while it:
-            path = self.get_path(it)
-            name = self[path][self.COL_NAME]
-            if name in packages:
-                self.include_item(path, binb="User Selected")
-                packages.remove(name)
-            it = self.iter_next(it)
-
-        self.emit("image-changed", rep.base_image)
-
-    """
-    squish lst so that it doesn't contain any duplicate entries
-    """
-    def squish(self, lst):
-        seen = {}
-        for l in lst:
-            seen[l] = 1
-        return seen.keys()
-
-    """
-    Mark the item at path as not included
-    NOTE:
-    path should be a gtk.TreeModelPath into self (not a filtered model)
-    """
-    def remove_item_path(self, path):
-        self[path][self.COL_BINB] = ""
-        self[path][self.COL_INC] = False
-
-    """
-    Recursively called to mark the item at opath and any package which
-    depends on it for removal.
-    NOTE: This method dumbly removes user selected packages and since we don't
-    do significant reverse dependency tracking it's easier and simpler to save
-    the items marked as user selected and re-add them once the removal sweep is
-    complete.
-    """
-    def mark(self, opath):
-        usersel = {}
-        removed = []
-
-        it = self.get_iter_first()
-        # The name of the item we're removing, so that we can use it to find
-        # other items which either depend on it, or were brought in by it
-        marked_name = self[opath][self.COL_NAME]
-
-        # Remove the passed item
-        self.remove_item_path(opath)
-
-        # Remove all dependent packages, update binb
-        while it:
-            path = self.get_path(it)
-            it = self.iter_next(it)
-
-            inc = self[path][self.COL_INC]
-            deps = self[path][self.COL_DEPS]
-            binb = self[path][self.COL_BINB].split(', ')
-            itype = self[path][self.COL_TYPE]
-            itname = self[path][self.COL_NAME]
-
-            # We ignore anything that isn't a package
-            if not itype == "package":
-                continue
-
-            # If the user added this item and it's not the item we're removing
-            # we should keep it and its dependencies, the easiest way to do so
-            # is to save its name and re-mark it for inclusion once dependency
-            # processing is complete
-            if "User Selected" in binb:
-                usersel[itname] = self[path][self.COL_IMG]
-
-            # If the iterated item is included and depends on the removed
-            # item it should also be removed.
-            # FIXME: need to ensure partial name matching doesn't happen
-            if inc and marked_name in deps and itname not in removed:
-                # found a dependency, remove it
-                removed.append(itname)
-                self.mark(path)
-
-            # If the iterated item was brought in by the removed (passed) item
-            # try and find an alternative dependee and update the binb column
-            if inc and marked_name in binb:
-                binb.remove(marked_name)
-                self[path][self.COL_BINB] = ', '.join(binb).lstrip(', ')
-
-        # Re-add any removed user selected items
-        for u in usersel:
-            npath = self.find_path_for_item(u)
-            self.include_item(item_path=npath,
-                              binb="User Selected",
-                              image_contents=usersel[u])
-    """
-    Remove items from contents if the have an empty COL_BINB (brought in by)
-    caused by all packages they are a dependency of being removed.
-    If the item isn't a package we leave it included.
-    """
-    def sweep_up(self):
-        it = self.contents.get_iter_first()
-        while it:
-            binb = self.contents.get_value(it, self.COL_BINB)
-            itype = self.contents.get_value(it, self.COL_TYPE)
-            remove = False
-
-            if itype == 'package' and not binb:
-                oit = self.contents.convert_iter_to_child_iter(it)
-                opath = self.get_path(oit)
-                self.mark(opath)
-                remove = True
-
-            # When we remove a package from the contents model we alter the
-            # model, so continuing to iterate is bad. *Furthermore* it's
-            # likely that the removal has affected an already iterated item
-            # so we should start from the beginning anyway.
-            # Only when we've managed to iterate the entire contents model
-            # without removing any items do we allow the loop to exit.
-            if remove:
-                it = self.contents.get_iter_first()
-            else:
-                it = self.contents.iter_next(it)
-
-    """
-    Check whether the item at item_path is included or not
-    """
-    def contents_includes_path(self, item_path):
-        return self[item_path][self.COL_INC]
-
-    """
-    Add this item, and any of its dependencies, to the image contents
-    """
-    def include_item(self, item_path, binb="", image_contents=False):
-        item_name = self[item_path][self.COL_NAME]
-        item_deps = self[item_path][self.COL_DEPS]
-
-        self[item_path][self.COL_INC] = True
-
-        item_bin = self[item_path][self.COL_BINB].split(', ')
-        if binb and not binb in item_bin:
-            item_bin.append(binb)
-            self[item_path][self.COL_BINB] = ', '.join(item_bin).lstrip(', ')
-
-        # We want to do some magic with things which are brought in by the
-        # base image so tag them as so
-        if image_contents:
-            self[item_path][self.COL_IMG] = True
-            if self[item_path][self.COL_TYPE] == 'image':
-                self.selected_image = item_name
-
-        if item_deps:
-            # Ensure all of the items deps are included and, where appropriate,
-            # add this item to their COL_BINB
-            for dep in item_deps.split(" "):
-                # If the contents model doesn't already contain dep, add it
-                dep_path = self.find_path_for_item(dep)
-                if not dep_path:
-                    continue
-                dep_included = self.contents_includes_path(dep_path)
-
-                if dep_included and not dep in item_bin:
-                    # don't set the COL_BINB to this item if the target is an
-                    # item in our own COL_BINB
-                    dep_bin = self[dep_path][self.COL_BINB].split(', ')
-                    if not item_name in dep_bin:
-                        dep_bin.append(item_name)
-                        self[dep_path][self.COL_BINB] = ', '.join(dep_bin).lstrip(', ')
-                elif not dep_included:
-                    self.include_item(dep_path, binb=item_name, image_contents=image_contents)
-
-    """
-    Find the model path for the item_name
-    Returns the path in the model or None
-    """
-    def find_path_for_item(self, item_name):
-        # We don't include virtual/* or *-native items in the model so save a
-        # heavy iteration loop by exiting early for these items
-        if self.non_target_name(item_name):
-            return None
-
-        it = self.get_iter_first()
-        while it:
-            if (self.get_value(it, self.COL_NAME) == item_name):
-                return self.get_path(it)
-            else:
-                it = self.iter_next(it)
-        return None
-
-    """
-    Empty self.contents by setting the include of each entry to None
-    """
-    def reset(self):
-        # Deselect images - slightly more complex logic so that we don't
-        # have to iterate all of the contents of the main model, instead
-        # just iterate the images model.
-        if self.selected_image:
-            iit = self.images.get_iter_first()
-            while iit:
-                pit = self.images.convert_iter_to_child_iter(iit)
-                self.set(pit, self.COL_INC, False)
-                iit = self.images.iter_next(iit)
-            self.selected_image = None
-
-        it = self.contents.get_iter_first()
-        while it:
-            oit = self.contents.convert_iter_to_child_iter(it)
-            self.set(oit,
-                     self.COL_INC, False,
-                     self.COL_BINB, "",
-                     self.COL_IMG, False)
-            # As we've just removed the first item...
-            it = self.contents.get_iter_first()
-
-    """
-    Returns two lists. One of user selected packages and the other containing
-    all selected packages
-    """
-    def get_selected_packages(self):
-        allpkgs = []
-        userpkgs = []
-
-        it = self.contents.get_iter_first()
-        while it:
-            sel = "User Selected" in self.contents.get_value(it, self.COL_BINB)
-            name = self.contents.get_value(it, self.COL_NAME)
-            allpkgs.append(name)
-            if sel:
-                userpkgs.append(name)
-            it = self.contents.iter_next(it)
-        return userpkgs, allpkgs
-
-    """
-    Return a squished (uniquified) list of the PN's of all selected items
-    """
-    def get_selected_pn(self):
-        pns = []
-
-        it = self.contents.get_iter_first()
-        while it:
-            if self.contents.get_value(it, self.COL_BINB):
-                pns.append(self.contents.get_value(it, self.COL_PN))
-            it = self.contents.iter_next(it)
-
-        return self.squish(pns)
-
-    def image_contents_removed(self):
-        it = self.get_iter_first()
-        while it:
-            sel = self.get_value(it, self.COL_INC)
-            img = self.get_value(it, self.COL_IMG)
-            if img and not sel:
-                return True
-            it = self.iter_next(it)
-        return False
-
-    def get_build_rep(self):
-        userpkgs, allpkgs = self.get_selected_packages()
-        # If base image contents have been removed start from an empty rootfs
-        if not self.selected_image or self.image_contents_removed():
-            image = "empty"
-        else:
-            image = self.selected_image
-
-        return BuildRep(" ".join(userpkgs), " ".join(allpkgs), image)
-
-    def find_reverse_depends(self, pn):
-        revdeps = []
-        it = self.contents.get_iter_first()
-
-        while it:
-            name = self.contents.get_value(it, self.COL_NAME)
-            itype = self.contents.get_value(it, self.COL_TYPE)
-            deps = self.contents.get_value(it, self.COL_DEPS)
-
-            it = self.contents.iter_next(it)
-
-            if not itype == 'package':
-                continue
-
-            if pn in deps:
-                revdeps.append(name)
-
-        if pn in revdeps:
-            revdeps.remove(pn)
-        return revdeps
-
-    def set_selected_image(self, img):
-        self.selected_image = img
-        path = self.find_path_for_item(img)
-        self.include_item(item_path=path,
-                          binb="User Selected",
-                          image_contents=True)
-
-        self.emit("image-changed", self.selected_image)
-
-    def set_selected_packages(self, pkglist):
-        selected = pkglist
-        it = self.get_iter_first()
-
-        while it:
-            name = self.get_value(it, self.COL_NAME)
-            if name in pkglist:
-                pkglist.remove(name)
-                path = self.get_path(it)
-                self.include_item(item_path=path,
-                                  binb="User Selected")
-                if len(pkglist) == 0:
-                    return
-            it = self.iter_next(it)
-
-    def find_image_path(self, image):
-        it = self.images.get_iter_first()
-
-        while it:
-            image_name = self.images.get_value(it, self.COL_NAME)
-            if image_name == image:
-                path = self.images.get_value(it, self.COL_PATH)
-                meta_pattern = "(\S*)/(meta*/)(\S*)"
-                meta_match = re.search(meta_pattern, path)
-                if meta_match:
-                    _, lyr, bbrel = path.partition(meta_match.group(2))
-                    if bbrel:
-                        path = bbrel
-                return path
-            it = self.images.iter_next(it)

bitbake/lib/bb/ui/depexp.py

@@ -19,12 +19,8 @@
 
 import gobject
 import gtk
-import Queue
 import threading
 import xmlrpclib
-import bb
-import bb.event
-from bb.ui.crumbs.progress import ProgressBar
 
 # Package Model
 (COL_PKG_NAME) = (0)
@@ -33,7 +29,6 @@ from bb.ui.crumbs.progress import ProgressBar
 (TYPE_DEP, TYPE_RDEP) = (0, 1)
 (COL_DEP_TYPE, COL_DEP_PARENT, COL_DEP_PACKAGE) = (0, 1, 2)
 
-
 class PackageDepView(gtk.TreeView):
     def __init__(self, model, dep_type, label):
         gtk.TreeView.__init__(self)
@@ -54,7 +49,6 @@ class PackageDepView(gtk.TreeView):
         self.current = package
         self.filter_model.refilter()
 
-
 class PackageReverseDepView(gtk.TreeView):
     def __init__(self, model, label):
         gtk.TreeView.__init__(self)
@@ -72,7 +66,6 @@ class PackageReverseDepView(gtk.TreeView):
         self.current = package
         self.filter_model.refilter()
 
-
 class DepExplorer(gtk.Window):
     def __init__(self):
         gtk.Window.__init__(self)
@@ -82,9 +75,7 @@ class DepExplorer(gtk.Window):
 
         # Create the data models
         self.pkg_model = gtk.ListStore(gobject.TYPE_STRING)
-        self.pkg_model.set_sort_column_id(COL_PKG_NAME, gtk.SORT_ASCENDING)
         self.depends_model = gtk.ListStore(gobject.TYPE_INT, gobject.TYPE_STRING, gobject.TYPE_STRING)
-        self.depends_model.set_sort_column_id(COL_DEP_PACKAGE, gtk.SORT_ASCENDING)
 
         pane = gtk.HPaned()
         pane.set_position(250)
@@ -94,11 +85,9 @@ class DepExplorer(gtk.Window):
         scrolled = gtk.ScrolledWindow()
         scrolled.set_policy(gtk.POLICY_AUTOMATIC, gtk.POLICY_AUTOMATIC)
         scrolled.set_shadow_type(gtk.SHADOW_IN)
-
         self.pkg_treeview = gtk.TreeView(self.pkg_model)
         self.pkg_treeview.get_selection().connect("changed", self.on_cursor_changed)
-        column = gtk.TreeViewColumn("Package", gtk.CellRendererText(), text=COL_PKG_NAME)
-        self.pkg_treeview.append_column(column)
+        self.pkg_treeview.append_column(gtk.TreeViewColumn("Package", gtk.CellRendererText(), text=COL_PKG_NAME))
         pane.add1(scrolled)
         scrolled.add(self.pkg_treeview)
 
@@ -164,6 +153,7 @@ class DepExplorer(gtk.Window):
 
 
 def parse(depgraph, pkg_model, depends_model):
+
     for package in depgraph["pn"]:
         pkg_model.set(pkg_model.append(), COL_PKG_NAME, package)
 
@@ -181,6 +171,17 @@ def parse(depgraph, pkg_model, depends_model):
                               COL_DEP_PARENT, package,
                               COL_DEP_PACKAGE, rdepend)
 
+class ProgressBar(gtk.Window):
+    def __init__(self):
+
+        gtk.Window.__init__(self)
+        self.set_title("Parsing .bb files, please wait...")
+        self.set_default_size(500, 0)
+        self.connect("delete-event", gtk.main_quit)
+
+        self.progress = gtk.ProgressBar()
+        self.add(self.progress)
+        self.show_all()
 
 class gtkthread(threading.Thread):
     quit = threading.Event()
@@ -195,17 +196,14 @@ class gtkthread(threading.Thread):
         gtk.main()
         gtkthread.quit.set()
 
+def init(server, eventHandler):
 
-def main(server, eventHandler):
     try:
         cmdline = server.runCommand(["getCmdLineAction"])
-        if cmdline and not cmdline['action']:
-            print(cmdline['msg'])
-            return
-        elif not cmdline or (cmdline['action'] and cmdline['action'][0] != "generateDotGraph"):
+        if not cmdline or cmdline[0] != "generateDotGraph":
             print("This UI is only compatible with the -g option")
             return
-        ret = server.runCommand(["generateDepTreeEvent", cmdline['action'][1], cmdline['action'][2]])
+        ret = server.runCommand(["generateDepTreeEvent", cmdline[1], cmdline[2]])
         if ret != True:
             print("Couldn't run command! %s" % ret)
             return
@@ -219,83 +217,46 @@ def main(server, eventHandler):
     gtkgui.start()
 
     gtk.gdk.threads_enter()
+    pbar = ProgressBar()
     dep = DepExplorer()
-    pbar = ProgressBar(dep)
-    pbar.connect("delete-event", gtk.main_quit)
     gtk.gdk.threads_leave()
 
-    progress_total = 0
     while True:
         try:
             event = eventHandler.waitEvent(0.25)
             if gtkthread.quit.isSet():
-                server.runCommand(["stateStop"])
                 break
 
             if event is None:
                 continue
-
-            if isinstance(event, bb.event.CacheLoadStarted):
-                progress_total = event.total
-                gtk.gdk.threads_enter()
-                pbar.set_title("Loading Cache")
-                pbar.update(0, progress_total)
-                gtk.gdk.threads_leave()
-
-            if isinstance(event, bb.event.CacheLoadProgress):
-                x = event.current
-                gtk.gdk.threads_enter()
-                pbar.update(x, progress_total)
-                gtk.gdk.threads_leave()
-                continue
-
-            if isinstance(event, bb.event.CacheLoadCompleted):
-                pbar.hide()
-                continue
-
-            if isinstance(event, bb.event.ParseStarted):
-                progress_total = event.total
-                if progress_total == 0:
-                    continue
-                gtk.gdk.threads_enter()
-                pbar.set_title("Processing recipes")
-                pbar.update(0, progress_total)
-                gtk.gdk.threads_leave()
-
             if isinstance(event, bb.event.ParseProgress):
-                x = event.current
+                x = event.sofar
+                y = event.total
+                if x == y:
+                    print(("\nParsing finished. %d cached, %d parsed, %d skipped, %d masked, %d errors."
+                        % ( event.cached, event.parsed, event.skipped, event.masked, event.errors)))
+                    pbar.hide()
                 gtk.gdk.threads_enter()
-                pbar.update(x, progress_total)
+                pbar.progress.set_fraction(float(x)/float(y))
+                pbar.progress.set_text("%d/%d (%2d %%)" % (x, y, x*100/y))
                 gtk.gdk.threads_leave()
                 continue
 
-            if isinstance(event, bb.event.ParseCompleted):
-                pbar.hide()
-                continue
-
             if isinstance(event, bb.event.DepTreeGenerated):
                 gtk.gdk.threads_enter()
                 parse(event._depgraph, dep.pkg_model, dep.depends_model)
                 gtk.gdk.threads_leave()
 
-            if isinstance(event, bb.command.CommandCompleted):
+            if isinstance(event, bb.command.CookerCommandCompleted):
                 continue
-
-            if isinstance(event, bb.command.CommandFailed):
+            if isinstance(event, bb.command.CookerCommandFailed):
                 print("Command execution failed: %s" % event.error)
-                return event.exitcode
-
-            if isinstance(event, bb.command.CommandExit):
-                return event.exitcode
-
+                break
             if isinstance(event, bb.cooker.CookerExit):
                 break
 
             continue
-        except EnvironmentError as ioerror:
-            # ignore interrupted io
-            if ioerror.args[0] == 4:
-                pass
+
         except KeyboardInterrupt:
             if shutdown == 2:
                 print("\nThird Keyboard Interrupt, exit.\n")

bitbake/lib/bb/ui/goggle.py

@@ -22,34 +22,17 @@ import gobject
 import gtk
 import xmlrpclib
 from bb.ui.crumbs.runningbuild import RunningBuildTreeView, RunningBuild
-from bb.ui.crumbs.progress import ProgressBar
 
-import Queue
-
-
-def event_handle_idle_func (eventHandler, build, pbar):
+def event_handle_idle_func (eventHandler, build):
 
     # Consume as many messages as we can in the time available to us
     event = eventHandler.getEvent()
     while event:
-        build.handle_event (event, pbar)
+        build.handle_event (event)
         event = eventHandler.getEvent()
 
     return True
 
-def scroll_tv_cb (model, path, iter, view):
-    view.scroll_to_cell (path)
-
-
-# @todo hook these into the GUI so the user has feedback...
-def running_build_failed_cb (running_build):
-    pass
-
-
-def running_build_succeeded_cb (running_build):
-    pass
-
-
 class MainWindow (gtk.Window):
     def __init__ (self):
         gtk.Window.__init__ (self, gtk.WINDOW_TOPLEVEL)
@@ -58,36 +41,24 @@ class MainWindow (gtk.Window):
         scrolled_window = gtk.ScrolledWindow ()
         self.add (scrolled_window)
         self.cur_build_tv = RunningBuildTreeView()
-        self.connect("delete-event", gtk.main_quit)
-        self.set_default_size(640, 480)
         scrolled_window.add (self.cur_build_tv)
 
-
-def main (server, eventHandler):
+def init (server, eventHandler):
     gobject.threads_init()
     gtk.gdk.threads_init()
 
     window = MainWindow ()
     window.show_all ()
-    pbar = ProgressBar(window)
-    pbar.connect("delete-event", gtk.main_quit)
 
     # Create the object for the current build
     running_build = RunningBuild ()
     window.cur_build_tv.set_model (running_build.model)
-    running_build.model.connect("row-inserted", scroll_tv_cb, window.cur_build_tv)
-    running_build.connect ("build-succeeded", running_build_succeeded_cb)
-    running_build.connect ("build-failed", running_build_failed_cb)
-
     try:
         cmdline = server.runCommand(["getCmdLineAction"])
+        print(cmdline)
         if not cmdline:
-            print("Nothing to do.  Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.")
             return 1
-        elif not cmdline['action']:
-            print(cmdline['msg'])
-            return 1
-        ret = server.runCommand(cmdline['action'])
+        ret = server.runCommand(cmdline)
         if ret != True:
             print("Couldn't get default commandline! %s" % ret)
             return 1
@@ -97,20 +68,9 @@ def main (server, eventHandler):
 
     # Use a timeout function for probing the event queue to find out if we
     # have a message waiting for us.
-    gobject.timeout_add (100,
+    gobject.timeout_add (200,
                          event_handle_idle_func,
                          eventHandler,
-                         running_build,
-                         pbar)
-
-    try:
-        gtk.main()
-    except EnvironmentError as ioerror:
-        # ignore interrupted io
-        if ioerror.args[0] == 4:
-            pass
-    except KeyboardInterrupt:
-        pass
-    finally:
-        server.runCommand(["stateStop"])
+                         running_build)
 
+    gtk.main()

bitbake/lib/bb/ui/knotty.py

@@ -22,49 +22,15 @@ from __future__ import division
 
 import os
 import sys
+import itertools
 import xmlrpclib
-import logging
-import progressbar
-import bb.msg
+from bb import ui
 from bb.ui import uihelper
 
-logger = logging.getLogger("BitBake")
-interactive = sys.stdout.isatty()
 
-class BBProgress(progressbar.ProgressBar):
-    def __init__(self, msg, maxval):
-        self.msg = msg
-        widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
-           progressbar.ETA()]
+parsespin = itertools.cycle( r'|/-\\' )
 
-        progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets)
-
-class NonInteractiveProgress(object):
-    fobj = sys.stdout
-
-    def __init__(self, msg, maxval):
-        self.msg = msg
-        self.maxval = maxval
-
-    def start(self):
-        self.fobj.write("%s..." % self.msg)
-        self.fobj.flush()
-        return self
-
-    def update(self, value):
-        pass
-
-    def finish(self):
-        self.fobj.write("done.\n")
-        self.fobj.flush()
-
-def new_progress(msg, maxval):
-    if interactive:
-        return BBProgress(msg, maxval)
-    else:
-        return NonInteractiveProgress(msg, maxval)
-
-def main(server, eventHandler):
+def init(server, eventHandler):
 
     # Get values of variables which control our output
     includelogs = server.runCommand(["getVariable", "BBINCLUDELOGS"])
@@ -72,21 +38,12 @@ def main(server, eventHandler):
 
     helper = uihelper.BBUIHelper()
 
-    console = logging.StreamHandler(sys.stdout)
-    format = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
-    bb.msg.addDefaultlogFilter(console)
-    console.setFormatter(format)
-    logger.addHandler(console)
-
     try:
         cmdline = server.runCommand(["getCmdLineAction"])
+        #print cmdline
         if not cmdline:
-            print("Nothing to do.  Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.")
             return 1
-        elif not cmdline['action']:
-            print(cmdline['msg'])
-            return 1
-        ret = server.runCommand(cmdline['action'])
+        ret = server.runCommand(cmdline)
         if ret != True:
             print("Couldn't get default commandline! %s" % ret)
             return 1
@@ -94,9 +51,6 @@ def main(server, eventHandler):
         print("XMLRPC Fault getting commandline:\n %s" % x)
         return 1
 
-
-    parseprogress = None
-    cacheprogress = None
     shutdown = 0
     return_value = 0
     while True:
@@ -104,6 +58,7 @@ def main(server, eventHandler):
             event = eventHandler.waitEvent(0.25)
             if event is None:
                 continue
+            #print event
             helper.eventHandler(event)
             if isinstance(event, bb.runqueue.runQueueExitWait):
                 if not shutdown:
@@ -112,21 +67,31 @@ def main(server, eventHandler):
                 activetasks, failedtasks = helper.getTasks()
                 if activetasks:
                     print("Waiting for %s active tasks to finish:" % len(activetasks))
-                    for tasknum, task in enumerate(activetasks):
+                    tasknum = 1
+                    for task in activetasks:
                         print("%s: %s (pid %s)" % (tasknum, activetasks[task]["title"], task))
+                        tasknum = tasknum + 1
 
-            if isinstance(event, logging.LogRecord):
-                if event.levelno >= format.ERROR:
-                    return_value = 1
-                # For "normal" logging conditions, don't show note logs from tasks
-                # but do show them if the user has changed the default log level to 
-                # include verbose/debug messages
-                #if logger.getEffectiveLevel() > format.VERBOSE:
-                if event.taskpid != 0 and event.levelno <= format.NOTE:
-                        continue
-                logger.handle(event)
+            if isinstance(event, bb.msg.MsgPlain):
+                print(event._message)
+                continue
+            if isinstance(event, bb.msg.MsgDebug):
+                print('DEBUG: ' + event._message)
+                continue
+            if isinstance(event, bb.msg.MsgNote):
+                print('NOTE: ' + event._message)
+                continue
+            if isinstance(event, bb.msg.MsgWarn):
+                print('WARNING: ' + event._message)
+                continue
+            if isinstance(event, bb.msg.MsgError):
+                return_value = 1
+                print('ERROR: ' + event._message)
+                continue
+            if isinstance(event, bb.msg.MsgFatal):
+                return_value = 1
+                print('FATAL: ' + event._message)
                 continue
-
             if isinstance(event, bb.build.TaskFailed):
                 return_value = 1
                 logfile = event.logfile
@@ -152,85 +117,53 @@ def main(server, eventHandler):
                             for line in lines:
                                 print(line)
             if isinstance(event, bb.build.TaskBase):
-                logger.info(event._message)
-                continue
-            if isinstance(event, bb.event.ParseStarted):
-                if event.total == 0:
-                    continue
-                parseprogress = new_progress("Parsing recipes", event.total).start()
+                print("NOTE: %s" % event._message)
                 continue
             if isinstance(event, bb.event.ParseProgress):
-                parseprogress.update(event.current)
-                continue
-            if isinstance(event, bb.event.ParseCompleted):
-                if not parseprogress:
-                    continue
-
-                parseprogress.finish()
-                print(("Parsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors."
-                    % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors)))
+                x = event.sofar
+                y = event.total
+                if os.isatty(sys.stdout.fileno()):
+                    sys.stdout.write("\rNOTE: Handling BitBake files: %s (%04d/%04d) [%2d %%]" % ( next(parsespin), x, y, x*100//y ) )
+                    sys.stdout.flush()
+                else:
+                    if x == 1:
+                        sys.stdout.write("Parsing .bb files, please wait...")
+                        sys.stdout.flush()
+                    if x == y:
+                        sys.stdout.write("done.")
+                        sys.stdout.flush()
+                if x == y:
+                    print(("\nParsing of %d .bb files complete (%d cached, %d parsed). %d targets, %d skipped, %d masked, %d errors."
+                        % ( event.total, event.cached, event.parsed, event.virtuals, event.skipped, event.masked, event.errors)))
                 continue
 
-            if isinstance(event, bb.event.CacheLoadStarted):
-                cacheprogress = new_progress("Loading cache", event.total).start()
-                continue
-            if isinstance(event, bb.event.CacheLoadProgress):
-                cacheprogress.update(event.current)
-                continue
-            if isinstance(event, bb.event.CacheLoadCompleted):
-                cacheprogress.finish()
-                print("Loaded %d entries from dependency cache." % event.num_entries)
-                continue
-
-            if isinstance(event, bb.command.CommandCompleted):
+            if isinstance(event, bb.command.CookerCommandCompleted):
                 break
-            if isinstance(event, bb.command.CommandFailed):
+            if isinstance(event, bb.command.CookerCommandSetExitCode):
                 return_value = event.exitcode
-                logger.error("Command execution failed: %s", event.error)
-                break
-            if isinstance(event, bb.command.CommandExit):
-                if not return_value:
-                    return_value = event.exitcode
                 continue
+            if isinstance(event, bb.command.CookerCommandFailed):
+                return_value = 1
+                print("Command execution failed: %s" % event.error)
+                break
             if isinstance(event, bb.cooker.CookerExit):
                 break
             if isinstance(event, bb.event.MultipleProviders):
-                logger.info("multiple providers are available for %s%s (%s)", event._is_runtime and "runtime " or "",
-                            event._item,
-                            ", ".join(event._candidates))
-                logger.info("consider defining a PREFERRED_PROVIDER entry to match %s", event._item)
+                print("NOTE: multiple providers are available for %s%s (%s)" % (event._is_runtime and "runtime " or "",
+                                                                                event._item,
+                                                                                ", ".join(event._candidates)))
+                print("NOTE: consider defining a PREFERRED_PROVIDER entry to match %s" % event._item)
                 continue
             if isinstance(event, bb.event.NoProvider):
-                return_value = 1
                 if event._runtime:
                     r = "R"
                 else:
                     r = ""
 
                 if event._dependees:
-                    logger.error("Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)", r, event._item, ", ".join(event._dependees), r)
+                    print("ERROR: Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)" % (r, event._item, ", ".join(event._dependees), r))
                 else:
-                    logger.error("Nothing %sPROVIDES '%s'", r, event._item)
-                if event._reasons:
-                    for reason in event._reasons:
-                        logger.error("%s", reason)
-                continue
-
-            if isinstance(event, bb.runqueue.runQueueTaskStarted):
-                if event.noexec:
-                    tasktype = 'noexec task'
-                else:
-                    tasktype = 'task'
-                logger.info("Running %s %s of %s (ID: %s, %s)",
-                            tasktype,
-                            event.stats.completed + event.stats.active +
-                                event.stats.failed + 1,
-                            event.stats.total, event.taskid, event.taskstring)
-                continue
-
-            if isinstance(event, bb.runqueue.runQueueTaskFailed):
-                logger.error("Task %s (%s) failed with exit code '%s'",
-                             event.taskid, event.taskstring, event.exitcode)
+                    print("ERROR: Nothing %sPROVIDES '%s'" % (r, event._item))
                 continue
 
             # ignore
@@ -238,17 +171,12 @@ def main(server, eventHandler):
                                   bb.event.StampUpdate,
                                   bb.event.ConfigParsed,
                                   bb.event.RecipeParsed,
-                                  bb.event.RecipePreFinalise,
                                   bb.runqueue.runQueueEvent,
                                   bb.runqueue.runQueueExitWait)):
                 continue
 
-            logger.error("Unknown event: %s", event)
+            print("Unknown Event: %s" % event)
 
-        except EnvironmentError as ioerror:
-            # ignore interrupted io
-            if ioerror.args[0] == 4:
-                pass
         except KeyboardInterrupt:
             if shutdown == 2:
                 print("\nThird Keyboard Interrupt, exit.\n")

bitbake/lib/bb/ui/ncurses.py

@@ -44,9 +44,8 @@
 
 """
 
-
 from __future__ import division
-import logging
+
 import os, sys, curses, itertools, time
 import bb
 import xmlrpclib
@@ -232,12 +231,8 @@ class NCursesUI:
         try:
             cmdline = server.runCommand(["getCmdLineAction"])
             if not cmdline:
-                print("Nothing to do.  Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.")
                 return
-            elif not cmdline['action']:
-                print(cmdline['msg'])
-                return
-            ret = server.runCommand(cmdline['action'])
+            ret = server.runCommand(cmdline)
             if ret != True:
                 print("Couldn't get default commandlind! %s" % ret)
                 return
@@ -251,39 +246,33 @@ class NCursesUI:
                 event = eventHandler.waitEvent(0.25)
                 if not event:
                     continue
-
                 helper.eventHandler(event)
+                #mw.appendText("%s\n" % event[0])
                 if isinstance(event, bb.build.TaskBase):
                     mw.appendText("NOTE: %s\n" % event._message)
-                if isinstance(event, logging.LogRecord):
-                    mw.appendText(logging.getLevelName(event.levelno) + ': ' + event.getMessage() + '\n')
-
-                if isinstance(event, bb.event.CacheLoadStarted):
-                    self.parse_total = event.total
-                if isinstance(event, bb.event.CacheLoadProgress):
-                    x = event.current
-                    y = self.parse_total
-                    mw.setStatus("Loading Cache:   %s [%2d %%]" % ( next(parsespin), x*100/y ) )
-                if isinstance(event, bb.event.CacheLoadCompleted):
-                    mw.setStatus("Idle")
-                    mw.appendText("Loaded %d entries from dependency cache.\n"
-                                % ( event.num_entries))
-
-                if isinstance(event, bb.event.ParseStarted):
-                    self.parse_total = event.total
+                if isinstance(event, bb.msg.MsgDebug):
+                    mw.appendText('DEBUG: ' + event._message + '\n')
+                if isinstance(event, bb.msg.MsgNote):
+                    mw.appendText('NOTE: ' + event._message + '\n')
+                if isinstance(event, bb.msg.MsgWarn):
+                    mw.appendText('WARNING: ' + event._message + '\n')
+                if isinstance(event, bb.msg.MsgError):
+                    mw.appendText('ERROR: ' + event._message + '\n')
+                if isinstance(event, bb.msg.MsgFatal):
+                    mw.appendText('FATAL: ' + event._message + '\n')
                 if isinstance(event, bb.event.ParseProgress):
-                    x = event.current
-                    y = self.parse_total
-                    mw.setStatus("Parsing Recipes: %s [%2d %%]" % ( next(parsespin), x*100/y ) )
-                if isinstance(event, bb.event.ParseCompleted):
-                    mw.setStatus("Idle")
-                    mw.appendText("Parsing finished. %d cached, %d parsed, %d skipped, %d masked.\n"
+                    x = event.sofar
+                    y = event.total
+                    if x == y:
+                        mw.setStatus("Idle")
+                        mw.appendText("Parsing finished. %d cached, %d parsed, %d skipped, %d masked."
                                 % ( event.cached, event.parsed, event.skipped, event.masked ))
-
+                    else:
+                        mw.setStatus("Parsing: %s (%04d/%04d) [%2d %%]" % ( next(parsespin), x, y, x*100//y ) )
 #                if isinstance(event, bb.build.TaskFailed):
 #                    if event.logfile:
 #                        if data.getVar("BBINCLUDELOGS", d):
-#                            bb.error("log data follows (%s)" % logfile)
+#                            bb.msg.error(bb.msg.domain.Build, "log data follows (%s)" % logfile)
 #                            number_of_lines = data.getVar("BBINCLUDELOGS_LINES", d)
 #                            if number_of_lines:
 #                                os.system('tail -n%s %s' % (number_of_lines, logfile))
@@ -297,18 +286,14 @@ class NCursesUI:
 #                                    print '| %s' % l
 #                                f.close()
 #                        else:
-#                            bb.error("see log in %s" % logfile)
+#                            bb.msg.error(bb.msg.domain.Build, "see log in %s" % logfile)
 
-                if isinstance(event, bb.command.CommandCompleted):
-                    # stop so the user can see the result of the build, but
-                    # also allow them to now exit with a single ^C
-                    shutdown = 2
-                if isinstance(event, bb.command.CommandFailed):
+                if isinstance(event, bb.command.CookerCommandCompleted):
+                    exitflag = True
+                if isinstance(event, bb.command.CookerCommandFailed):
                     mw.appendText("Command execution failed: %s" % event.error)
                     time.sleep(2)
                     exitflag = True
-                if isinstance(event, bb.command.CommandExit):
-                    exitflag = True
                 if isinstance(event, bb.cooker.CookerExit):
                     exitflag = True
 
@@ -319,18 +304,13 @@ class NCursesUI:
                     if activetasks:
                         taw.appendText("Active Tasks:\n")
                         for task in activetasks.itervalues():
-                            taw.appendText(task["title"] + '\n')
+                            taw.appendText(task["title"])
                     if failedtasks:
                         taw.appendText("Failed Tasks:\n")
                         for task in failedtasks:
-                            taw.appendText(task["title"] + '\n')
+                            taw.appendText(task["title"])
 
                 curses.doupdate()
-            except EnvironmentError as ioerror:
-                # ignore interrupted io
-                if ioerror.args[0] == 4:
-                    pass
-
             except KeyboardInterrupt:
                 if shutdown == 2:
                     mw.appendText("Third Keyboard Interrupt, exit.\n")
@@ -344,7 +324,7 @@ class NCursesUI:
                 shutdown = shutdown + 1
                 pass
 
-def main(server, eventHandler):
+def init(server, eventHandler):
     if not os.isatty(sys.stdout.fileno()):
         print("FATAL: Unable to run 'ncurses' UI without a TTY.")
         return

bitbake/lib/bb/ui/puccho.py

@@ -390,7 +390,7 @@ def running_build_failed_cb (running_build, manager):
     print("build failed")
     manager.notify_build_failed ()
 
-def main (server, eventHandler):
+def init (server, eventHandler):
     # Initialise threading...
     gobject.threads_init()
     gtk.gdk.threads_init()

bitbake/lib/bb/ui/uievent.py

@@ -37,8 +37,8 @@ class BBUIEventQueue:
         self.BBServer = BBServer
 
         self.t = threading.Thread()
-        self.t.setDaemon(True)
-        self.t.run = self.startCallbackHandler
+        self.t.setDaemon(True)
+        self.t.run = self.startCallbackHandler
         self.t.start()
 
     def getEvent(self):
@@ -63,20 +63,17 @@ class BBUIEventQueue:
 
     def queue_event(self, event):
         self.eventQueueLock.acquire()
-        self.eventQueue.append(event)
+        self.eventQueue.append(pickle.loads(event))
         self.eventQueueNotify.set()
         self.eventQueueLock.release()
 
-    def send_event(self, event):
-        self.queue_event(pickle.loads(event))
-
     def startCallbackHandler(self):
 
         server = UIXMLRPCServer()
-        self.host, self.port = server.socket.getsockname()
+        self.host, self.port = server.socket.getsockname()
 
         server.register_function( self.system_quit, "event.quit" )
-        server.register_function( self.send_event, "event.sendpickle" )
+        server.register_function( self.queue_event, "event.send" )
         server.socket.settimeout(1)
 
         self.EventHandle = self.BBServer.registerEventHandler(self.host, self.port)
@@ -86,7 +83,7 @@ class BBUIEventQueue:
             server.handle_request()
         server.server_close()
 
-    def system_quit( self ):
+    def system_quit( self ):
         """
         Shut down the callback thread
         """
@@ -98,11 +95,11 @@ class BBUIEventQueue:
 
 class UIXMLRPCServer (SimpleXMLRPCServer):
 
-    def __init__( self, interface = ("localhost", 0) ):
+    def __init__( self, interface = ("localhost", 0) ):
         self.quit = False
-        SimpleXMLRPCServer.__init__( self,
-                                    interface,
-                                    requestHandler=SimpleXMLRPCRequestHandler,
+        SimpleXMLRPCServer.__init__( self,
+                                    interface,
+                                    requestHandler=SimpleXMLRPCRequestHandler,
                                     logRequests=False, allow_none=True)
 
     def get_request(self):
@@ -124,4 +121,4 @@ class UIXMLRPCServer (SimpleXMLRPCServer):
         if request is None:
             return
         SimpleXMLRPCServer.process_request(self, request, client_address)
-
+

bitbake/lib/bb/ui/uihelper.py

@@ -17,8 +17,6 @@
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
-import bb.build
-
 class BBUIHelper:
     def __init__(self):
         self.needUpdate = False
@@ -37,6 +35,16 @@ class BBUIHelper:
             self.failed_tasks.append( { 'title' : "%s %s" % (event._package, event._task)})
             self.needUpdate = True
 
+        # Add runqueue event handling
+        #if isinstance(event, bb.runqueue.runQueueTaskCompleted):
+        #    a = 1
+        #if isinstance(event, bb.runqueue.runQueueTaskStarted):
+        #    a = 1
+        #if isinstance(event, bb.runqueue.runQueueTaskFailed):
+        #    a = 1
+        #if isinstance(event, bb.runqueue.runQueueExitWait):
+        #    a = 1
+
     def getTasks(self):
         self.needUpdate = False
         return (self.running_tasks, self.failed_tasks)

bitbake/lib/bb/utils.py

@@ -21,14 +21,10 @@ BitBake Utility Functions
 
 import re, fcntl, os, string, stat, shutil, time
 import sys
-import errno
-import logging
 import bb
+import errno
 import bb.msg
 from commands import getstatusoutput
-from contextlib import contextmanager
-
-logger = logging.getLogger("BitBake.Util")
 
 # Version comparison
 separators = ".-"
@@ -94,7 +90,7 @@ def vercmp(ta, tb):
     (ea, va, ra) = ta
     (eb, vb, rb) = tb
 
-    r = int(ea or 0) - int(eb or 0)
+    r = int(ea)-int(eb)
     if (r == 0):
         r = vercmp_part(va, vb)
     if (r == 0):
@@ -195,10 +191,10 @@ def vercmp_string(val1, val2):
     val2 = val2[0].split('.')
 
     # add back decimal point so that .03 does not become "3" !
-    for x in xrange(1, len(val1)):
+    for x in range(1, len(val1)):
         if val1[x][0] == '0' :
             val1[x] = '.' + val1[x]
-    for x in xrange(1, len(val2)):
+    for x in range(1, len(val2)):
         if val2[x][0] == '0' :
             val2[x] = '.' + val2[x]
 
@@ -215,10 +211,10 @@ def vercmp_string(val1, val2):
         val2[-1] += '_' + val2_prepart
     # The above code will extend version numbers out so they
     # have the same number of digits.
-    for x in xrange(0, len(val1)):
+    for x in range(0, len(val1)):
         cmp1 = relparse(val1[x])
         cmp2 = relparse(val2[x])
-        for y in xrange(0, 3):
+        for y in range(0, 3):
             myret = cmp1[y] - cmp2[y]
             if myret != 0:
                 __vercmp_cache__[valkey] = myret
@@ -279,7 +275,7 @@ def explode_dep_versions(s):
 
     return r
 
-def join_deps(deps, commasep=True):
+def join_deps(deps):
     """
     Take the result from explode_dep_versions and generate a dependency string
     """
@@ -289,10 +285,18 @@ def join_deps(deps, commasep=True):
             result.append(dep + " (" + deps[dep] + ")")
         else:
             result.append(dep)
-    if commasep:
-        return ", ".join(result)
-    else:
-        return " ".join(result)
+    return ", ".join(result)
+
+def extend_deps(dest, src):
+    """
+    Extend the results from explode_dep_versions by appending all of the items
+    in the second list, avoiding duplicates.
+    """
+    for dep in src:
+        if dep not in dest:
+            dest[dep] = src[dep]
+        elif dest[dep] != src[dep]:
+            dest[dep] = src[dep]
 
 def _print_trace(body, line):
     """
@@ -300,12 +304,10 @@ def _print_trace(body, line):
     """
     # print the environment of the method
     min_line = max(1, line-4)
-    max_line = min(line + 4, len(body))
-    for i in xrange(min_line, max_line + 1):
-        if line == i:
-            logger.error(' *** %.4d:%s', i, body[i-1])
-        else:
-            logger.error('     %.4d:%s', i, body[i-1])
+    max_line = min(line + 4, len(body)-1)
+    for i in range(min_line, max_line + 1):
+        bb.msg.error(bb.msg.domain.Util, "\t%.4d:%s" % (i, body[i-1]) )
+
 
 def better_compile(text, file, realfile, mode = "exec"):
     """
@@ -317,69 +319,50 @@ def better_compile(text, file, realfile, mode = "exec"):
     except Exception as e:
         # split the text into lines again
         body = text.split('\n')
-        logger.error("Error in compiling python function in %s", realfile)
-        logger.error(str(e))
+        bb.msg.error(bb.msg.domain.Util, "Error in compiling python function in: %s" % (realfile))
+        bb.msg.error(bb.msg.domain.Util, str(e))
         if e.lineno:
-            logger.error("The lines leading to this error were:")
-            logger.error("\t%d:%s:'%s'", e.lineno, e.__class__.__name__, body[e.lineno-1])
+            bb.msg.error(bb.msg.domain.Util, "The lines leading to this error were:")
+            bb.msg.error(bb.msg.domain.Util, "\t%d:%s:'%s'" % (e.lineno, e.__class__.__name__, body[e.lineno-1]))
             _print_trace(body, e.lineno)
         else:
-            logger.error("The function causing this error was:")
+            bb.msg.error(bb.msg.domain.Util, "The function causing this error was:")
             for line in body:
-                logger.error(line)
-
+                bb.msg.error(bb.msg.domain.Util, line)
         raise
 
-def better_exec(code, context, text, realfile = "<code>"):
+def better_exec(code, context, text, realfile):
     """
     Similiar to better_compile, better_exec will
     print the lines that are responsible for the
     error.
     """
     import bb.parse
-    if not hasattr(code, "co_filename"):
-        code = better_compile(code, realfile, realfile)
     try:
         exec(code, _context, context)
-    except Exception:
+    except:
         (t, value, tb) = sys.exc_info()
 
         if t in [bb.parse.SkipPackage, bb.build.FuncFailed]:
             raise
 
-        import traceback
-        exception = traceback.format_exception_only(t, value)
-        logger.error('Error executing a python function in %s:\n%s',
-                     realfile, ''.join(exception))
+        # print the Header of the Error Message
+        bb.msg.error(bb.msg.domain.Util, "Error in executing python function in: %s" % realfile)
+        bb.msg.error(bb.msg.domain.Util, "Exception:%s Message:%s" % (t, value))
 
         # Strip 'us' from the stack (better_exec call)
         tb = tb.tb_next
 
-        textarray = text.split('\n')
-        linefailed = traceback.tb_lineno(tb)
-
+        import traceback
         tbextract = traceback.extract_tb(tb)
-        tbformat = "\n".join(traceback.format_list(tbextract))
-        logger.error("The stack trace of python calls that resulted in this exception/failure was:")
-        for line in tbformat.split('\n'):
-            logger.error(line)
+        tbextract = "\n".join(traceback.format_list(tbextract))
+        bb.msg.error(bb.msg.domain.Util, "Traceback:")
+        for line in tbextract.split('\n'):
+            bb.msg.error(bb.msg.domain.Util, line)
 
-        logger.error("The code that was being executed was:")
-        _print_trace(textarray, linefailed)
-        logger.error("(file: '%s', lineno: %s, function: %s)", tbextract[0][0], tbextract[0][1], tbextract[0][2])
-
-        # See if this is a function we constructed and has calls back into other functions in
-        # "text". If so, try and improve the context of the error by diving down the trace
-        level = 0
-        nexttb = tb.tb_next
-        while nexttb is not None:
-            if tbextract[level][0] == tbextract[level+1][0] and tbextract[level+1][2] == tbextract[level][0]:
-                _print_trace(textarray, tbextract[level+1][1])
-                logger.error("(file: '%s', lineno: %s, function: %s)", tbextract[level+1][0], tbextract[level+1][1], tbextract[level+1][2])
-            else:
-                 break
-            nexttb = tb.tb_next
-            level = level + 1
+        line = traceback.tb_lineno(tb)
+        bb.msg.error(bb.msg.domain.Util, "The lines leading to this error were:")
+        _print_trace( text.split('\n'), line )
 
         raise
 
@@ -389,38 +372,16 @@ def simple_exec(code, context):
 def better_eval(source, locals):
     return eval(source, _context, locals)
 
-@contextmanager
-def fileslocked(files):
-    """Context manager for locking and unlocking file locks."""
-    locks = []
-    if files:
-        for lockfile in files:
-            locks.append(bb.utils.lockfile(lockfile))
-
-    yield
-
-    for lock in locks:
-        bb.utils.unlockfile(lock)
-
-def lockfile(name, shared=False, retry=True):
+def lockfile(name):
     """
     Use the file fn as a lock file, return when the lock has been acquired.
     Returns a variable to pass to unlockfile().
     """
-    dirname = os.path.dirname(name)
-    mkdirhier(dirname)
-
-    if not os.access(dirname, os.W_OK):
-        logger.error("Unable to acquire lock '%s', directory is not writable",
-                     name)
+    path = os.path.dirname(name)
+    if not os.path.isdir(path):
+        bb.msg.error(bb.msg.domain.Util, "Error, lockfile path does not exist!: %s" % path)
         sys.exit(1)
 
-    op = fcntl.LOCK_EX
-    if shared:
-        op = fcntl.LOCK_SH
-    if not retry:
-        op = op | fcntl.LOCK_NB
-
     while True:
         # If we leave the lockfiles lying around there is no problem
         # but we should clean up after ourselves. This gives potential
@@ -433,33 +394,25 @@ def lockfile(name, shared=False, retry=True):
         # lock is the most likely to win it.
 
         try:
-            lf = open(name, 'a+')
-            fileno = lf.fileno()
-            fcntl.flock(fileno, op)
-            statinfo = os.fstat(fileno)
+            lf = open(name, "a + ")
+            fcntl.flock(lf.fileno(), fcntl.LOCK_EX)
+            statinfo = os.fstat(lf.fileno())
             if os.path.exists(lf.name):
                 statinfo2 = os.stat(lf.name)
                 if statinfo.st_ino == statinfo2.st_ino:
                     return lf
-            lf.close()
-        except Exception:
-            pass
-        if not retry:
-            return None
+            # File no longer exists or changed, retry
+            lf.close
+        except Exception as e:
+            continue
 
 def unlockfile(lf):
     """
     Unlock a file locked using lockfile()
     """
-    try:
-        # If we had a shared lock, we need to promote to exclusive before
-        # removing the lockfile. Attempt this, ignore failures.
-        fcntl.flock(lf.fileno(), fcntl.LOCK_EX|fcntl.LOCK_NB)
-        os.unlink(lf.name)
-    except (IOError, OSError):
-        pass
+    os.unlink(lf.name)
     fcntl.flock(lf.fileno(), fcntl.LOCK_UN)
-    lf.close()
+    lf.close
 
 def md5_file(filename):
     """
@@ -493,24 +446,13 @@ def sha256_file(filename):
         s.update(line)
     return s.hexdigest()
 
-def preserved_envvars_exported():
-    """Variables which are taken from the environment and placed in and exported
-    from the metadata"""
+def preserved_envvars_list():
     return [
+        'BBPATH',
+        'BB_PRESERVE_ENV',
+        'BB_ENV_WHITELIST',
+        'BB_ENV_EXTRAWHITE',
         'BB_TASKHASH',
-        'HOME',
-        'LOGNAME',
-        'PATH',
-        'PWD',
-        'SHELL',
-        'TERM',
-        'USER',
-    ]
-
-def preserved_envvars_exported_interactive():
-    """Variables which are taken from the environment and placed in and exported
-    from the metadata, for interactive tasks"""
-    return [
         'COLORTERM',
         'DBUS_SESSION_BUS_ADDRESS',
         'DESKTOP_SESSION',
@@ -520,26 +462,23 @@ def preserved_envvars_exported_interactive():
         'GNOME_KEYRING_SOCKET',
         'GPG_AGENT_INFO',
         'GTK_RC_FILES',
+        'HOME',
+        'LANG',
+        'LOGNAME',
+        'PATH',
+        'PWD',
         'SESSION_MANAGER',
-        'KRB5CCNAME',
+        'SHELL',
         'SSH_AUTH_SOCK',
+        'TERM',
+        'USER',
+        'USERNAME',
+        '_',
         'XAUTHORITY',
         'XDG_DATA_DIRS',
         'XDG_SESSION_COOKIE',
     ]
 
-def preserved_envvars():
-    """Variables which are taken from the environment and placed in the metadata"""
-    v = [
-        'BBPATH',
-        'BB_PRESERVE_ENV',
-        'BB_ENV_WHITELIST',
-        'BB_ENV_EXTRAWHITE',
-        'LANG',
-        '_',
-    ]
-    return v + preserved_envvars_exported() + preserved_envvars_exported_interactive()
-
 def filter_environment(good_vars):
     """
     Create a pristine environment for bitbake. This will remove variables that
@@ -556,35 +495,22 @@ def filter_environment(good_vars):
         del os.environ[key]
 
     if len(removed_vars):
-        logger.debug(1, "Removed the following variables from the environment: %s", ", ".join(removed_vars))
+        bb.msg.debug(1, bb.msg.domain.Util, "Removed the following variables from the environment: %s" % (", ".join(removed_vars)))
 
     return removed_vars
 
-def create_interactive_env(d):
-    for k in preserved_envvars_exported_interactive():
-        os.setenv(k, bb.data.getVar(k, d, True))
-
-def approved_variables():
-    """
-    Determine and return the list of whitelisted variables which are approved
-    to remain in the envrionment.
-    """
-    approved = []
-    if 'BB_ENV_WHITELIST' in os.environ:
-        approved = os.environ['BB_ENV_WHITELIST'].split()
-    else:
-        approved = preserved_envvars()
-    if 'BB_ENV_EXTRAWHITE' in os.environ:
-        approved.extend(os.environ['BB_ENV_EXTRAWHITE'].split())
-    return approved
-
 def clean_environment():
     """
     Clean up any spurious environment variables. This will remove any
-    variables the user hasn't chosen to preserve.
+    variables the user hasn't chose to preserve.
     """
     if 'BB_PRESERVE_ENV' not in os.environ:
-        good_vars = approved_variables()
+        if 'BB_ENV_WHITELIST' in os.environ:
+            good_vars = os.environ['BB_ENV_WHITELIST'].split()
+        else:
+            good_vars = preserved_envvars_list()
+        if 'BB_ENV_EXTRAWHITE' in os.environ:
+            good_vars.extend(os.environ['BB_ENV_EXTRAWHITE'].split())
         filter_environment(good_vars)
 
 def empty_environment():
@@ -605,20 +531,6 @@ def build_environment(d):
         if export:
             os.environ[var] = bb.data.getVar(var, d, True) or ""
 
-def remove(path, recurse=False):
-    """Equivalent to rm -f or rm -rf"""
-    if not path:
-        return
-    import os, errno, shutil, glob
-    for name in glob.glob(path):
-        try:
-            os.unlink(name)
-        except OSError as exc:
-            if recurse and exc.errno == errno.EISDIR:
-                shutil.rmtree(name)
-            elif exc.errno != errno.ENOENT:
-                raise
-
 def prunedir(topdir):
     # Delete everything reachable from the directory named in 'topdir'.
     # CAUTION:  This is dangerous!
@@ -644,13 +556,15 @@ def prune_suffix(var, suffixes, d):
             return var.replace(suffix, "")
     return var
 
-def mkdirhier(directory):
+def mkdirhier(dir):
     """Create a directory like 'mkdir -p', but does not complain if
     directory already exists like os.makedirs
     """
 
+    bb.msg.debug(3, bb.msg.domain.Util, "mkdirhier(%s)" % dir)
     try:
-        os.makedirs(directory)
+        os.makedirs(dir)
+        bb.msg.debug(2, bb.msg.domain.Util, "created " + dir)
     except OSError as e:
         if e.errno != errno.EEXIST:
             raise e
@@ -784,23 +698,16 @@ def copyfile(src, dest, newmtime = None, sstat = None):
             return False
 
     if stat.S_ISREG(sstat[stat.ST_MODE]):
-        try:
-            srcchown = False
-            if not os.access(src, os.R_OK):
-                # Make sure we can read it
-                srcchown = True
-                os.chmod(src, sstat[stat.ST_MODE] | stat.S_IRUSR)
-
-            # For safety copy then move it over.
+        os.chmod(src, stat.S_IRUSR) # Make sure we can read it
+        try: # For safety copy then move it over.
             shutil.copyfile(src, dest + "#new")
             os.rename(dest + "#new", dest)
         except Exception as e:
             print('copyfile: copy', src, '->', dest, 'failed.', e)
             return False
         finally:
-            if srcchown:
-                os.chmod(src, sstat[stat.ST_MODE])
-                os.utime(src, (sstat[stat.ST_ATIME], sstat[stat.ST_MTIME]))
+            os.chmod(src, sstat[stat.ST_MODE])
+            os.utime(src, (sstat[stat.ST_ATIME], sstat[stat.ST_MTIME]))
 
     else:
         #we don't yet handle special, so we need to fall back to /bin/mv
@@ -838,27 +745,18 @@ def which(path, item, direction = 0):
 
     return ""
 
-def to_boolean(string, default=None):
-    if not string:
-        return default
+def init_logger(logger, verbose, debug, debug_domains):
+    """
+    Set verbosity and debug levels in the logger
+    """
 
-    normalized = string.lower()
-    if normalized in ("y", "yes", "1", "true"):
-        return True
-    elif normalized in ("n", "no", "0", "false"):
-        return False
-    else:
-        raise ValueError("Invalid value for to_boolean: %s" % string)
+    if verbose:
+        logger.set_verbose(True)
 
-def contains(variable, checkvalues, truevalue, falsevalue, d):
-    val = d.getVar(variable, True)
-    if not val:
-        return falsevalue
-    val = set(val.split())
-    if isinstance(checkvalues, basestring):
-        checkvalues = set(checkvalues.split())
+    if debug:
+        logger.set_debug_level(debug)
     else:
-        checkvalues = set(checkvalues)
-    if checkvalues.issubset(val): 
-        return truevalue
-    return falsevalue
+        logger.set_debug_level(0)
+
+    if debug_domains:
+        logger.set_debug_domains(debug_domains)

bitbake/lib/progressbar.py

@@ -1,384 +0,0 @@
-#!/usr/bin/python
-# -*- coding: iso-8859-1 -*-
-#
-# progressbar  - Text progressbar library for python.
-# Copyright (c) 2005 Nilton Volpato
-#
-# This library is free software; you can redistribute it and/or
-# modify it under the terms of the GNU Lesser General Public
-# License as published by the Free Software Foundation; either
-# version 2.1 of the License, or (at your option) any later version.
-#
-# This library is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-# Lesser General Public License for more details.
-#
-# You should have received a copy of the GNU Lesser General Public
-# License along with this library; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
-
-
-"""Text progressbar library for python.
-
-This library provides a text mode progressbar. This is typically used
-to display the progress of a long running operation, providing a
-visual clue that processing is underway.
-
-The ProgressBar class manages the progress, and the format of the line
-is given by a number of widgets. A widget is an object that may
-display diferently depending on the state of the progress. There are
-three types of widget:
-- a string, which always shows itself;
-- a ProgressBarWidget, which may return a diferent value every time
-it's update method is called; and
-- a ProgressBarWidgetHFill, which is like ProgressBarWidget, except it
-expands to fill the remaining width of the line.
-
-The progressbar module is very easy to use, yet very powerful. And
-automatically supports features like auto-resizing when available.
-"""
-
-from __future__ import division
-
-__author__ = "Nilton Volpato"
-__author_email__ = "first-name dot last-name @ gmail.com"
-__date__ = "2006-05-07"
-__version__ = "2.3-dev"
-
-import sys, time, os
-from array import array
-try:
-    from fcntl import ioctl
-    import termios
-except ImportError:
-    pass
-import signal
-try:
-    basestring
-except NameError:
-    basestring = (str,)
-
-class ProgressBarWidget(object):
-    """This is an element of ProgressBar formatting.
-
-    The ProgressBar object will call it's update value when an update
-    is needed. It's size may change between call, but the results will
-    not be good if the size changes drastically and repeatedly.
-    """
-    def update(self, pbar):
-        """Returns the string representing the widget.
-
-        The parameter pbar is a reference to the calling ProgressBar,
-        where one can access attributes of the class for knowing how
-        the update must be made.
-
-        At least this function must be overriden."""
-        pass
-
-class ProgressBarWidgetHFill(object):
-    """This is a variable width element of ProgressBar formatting.
-
-    The ProgressBar object will call it's update value, informing the
-    width this object must the made. This is like TeX \\hfill, it will
-    expand to fill the line. You can use more than one in the same
-    line, and they will all have the same width, and together will
-    fill the line.
-    """
-    def update(self, pbar, width):
-        """Returns the string representing the widget.
-
-        The parameter pbar is a reference to the calling ProgressBar,
-        where one can access attributes of the class for knowing how
-        the update must be made. The parameter width is the total
-        horizontal width the widget must have.
-
-        At least this function must be overriden."""
-        pass
-
-
-class ETA(ProgressBarWidget):
-    "Widget for the Estimated Time of Arrival"
-    def format_time(self, seconds):
-        return time.strftime('%H:%M:%S', time.gmtime(seconds))
-    def update(self, pbar):
-        if pbar.currval == 0:
-            return 'ETA:  --:--:--'
-        elif pbar.finished:
-            return 'Time: %s' % self.format_time(pbar.seconds_elapsed)
-        else:
-            elapsed = pbar.seconds_elapsed
-            eta = elapsed * pbar.maxval / pbar.currval - elapsed
-            return 'ETA:  %s' % self.format_time(eta)
-
-class FileTransferSpeed(ProgressBarWidget):
-    "Widget for showing the transfer speed (useful for file transfers)."
-    def __init__(self, unit='B'):
-        self.unit = unit
-        self.fmt = '%6.2f %s'
-        self.prefixes = ['', 'K', 'M', 'G', 'T', 'P']
-    def update(self, pbar):
-        if pbar.seconds_elapsed < 2e-6:#== 0:
-            bps = 0.0
-        else:
-            bps = pbar.currval / pbar.seconds_elapsed
-        spd = bps
-        for u in self.prefixes:
-            if spd < 1000:
-                break
-            spd /= 1000
-        return self.fmt % (spd, u + self.unit + '/s')
-
-class RotatingMarker(ProgressBarWidget):
-    "A rotating marker for filling the bar of progress."
-    def __init__(self, markers='|/-\\'):
-        self.markers = markers
-        self.curmark = -1
-    def update(self, pbar):
-        if pbar.finished:
-            return self.markers[0]
-        self.curmark = (self.curmark + 1) % len(self.markers)
-        return self.markers[self.curmark]
-
-class Percentage(ProgressBarWidget):
-    "Just the percentage done."
-    def update(self, pbar):
-        return '%3d%%' % pbar.percentage()
-
-class SimpleProgress(ProgressBarWidget):
-    "Returns what is already done and the total, e.g.: '5 of 47'"
-    def __init__(self, sep=' of '):
-        self.sep = sep
-    def update(self, pbar):
-        return '%d%s%d' % (pbar.currval, self.sep, pbar.maxval)
-
-class Bar(ProgressBarWidgetHFill):
-    "The bar of progress. It will stretch to fill the line."
-    def __init__(self, marker='#', left='|', right='|'):
-        self.marker = marker
-        self.left = left
-        self.right = right
-    def _format_marker(self, pbar):
-        if isinstance(self.marker, basestring):
-            return self.marker
-        else:
-            return self.marker.update(pbar)
-    def update(self, pbar, width):
-        percent = pbar.percentage()
-        cwidth = width - len(self.left) - len(self.right)
-        marked_width = int(percent * cwidth // 100)
-        m = self._format_marker(pbar)
-        bar = (self.left + (m * marked_width).ljust(cwidth) + self.right)
-        return bar
-
-class ReverseBar(Bar):
-    "The reverse bar of progress, or bar of regress. :)"
-    def update(self, pbar, width):
-        percent = pbar.percentage()
-        cwidth = width - len(self.left) - len(self.right)
-        marked_width = int(percent * cwidth // 100)
-        m = self._format_marker(pbar)
-        bar = (self.left + (m*marked_width).rjust(cwidth) + self.right)
-        return bar
-
-default_widgets = [Percentage(), ' ', Bar()]
-class ProgressBar(object):
-    """This is the ProgressBar class, it updates and prints the bar.
-
-    A common way of using it is like:
-    >>> pbar = ProgressBar().start()
-    >>> for i in xrange(100):
-    ...    # do something
-    ...    pbar.update(i+1)
-    ...
-    >>> pbar.finish()
-
-    You can also use a progressbar as an iterator:
-    >>> progress = ProgressBar()
-    >>> for i in progress(some_iterable):
-    ...    # do something
-    ...
-
-    But anything you want to do is possible (well, almost anything).
-    You can supply different widgets of any type in any order. And you
-    can even write your own widgets! There are many widgets already
-    shipped and you should experiment with them.
-
-    The term_width parameter must be an integer or None. In the latter case
-    it will try to guess it, if it fails it will default to 80 columns.
-
-    When implementing a widget update method you may access any
-    attribute or function of the ProgressBar object calling the
-    widget's update method. The most important attributes you would
-    like to access are:
-    - currval: current value of the progress, 0 <= currval <= maxval
-    - maxval: maximum (and final) value of the progress
-    - finished: True if the bar has finished (reached 100%), False o/w
-    - start_time: the time when start() method of ProgressBar was called
-    - seconds_elapsed: seconds elapsed since start_time
-    - percentage(): percentage of the progress [0..100]. This is a method.
-
-    The attributes above are unlikely to change between different versions,
-    the other ones may change or cease to exist without notice, so try to rely
-    only on the ones documented above if you are extending the progress bar.
-    """
-
-    __slots__ = ('currval', 'fd', 'finished', 'last_update_time', 'maxval',
-                 'next_update', 'num_intervals', 'seconds_elapsed',
-                 'signal_set', 'start_time', 'term_width', 'update_interval',
-                 'widgets', '_iterable')
-
-    _DEFAULT_MAXVAL = 100
-
-    def __init__(self, maxval=None, widgets=default_widgets, term_width=None,
-                 fd=sys.stderr):
-        self.maxval = maxval
-        self.widgets = widgets
-        self.fd = fd
-        self.signal_set = False
-        if term_width is not None:
-            self.term_width = term_width
-        else:
-            try:
-                self._handle_resize(None, None)
-                signal.signal(signal.SIGWINCH, self._handle_resize)
-                self.signal_set = True
-            except (SystemExit, KeyboardInterrupt):
-                raise
-            except:
-                self.term_width = int(os.environ.get('COLUMNS', 80)) - 1
-
-        self.currval = 0
-        self.finished = False
-        self.start_time = None
-        self.last_update_time = None
-        self.seconds_elapsed = 0
-        self._iterable = None
-
-    def __call__(self, iterable):
-        try:
-            self.maxval = len(iterable)
-        except TypeError:
-            # If the iterable has no length, then rely on the value provided
-            # by the user, otherwise fail.
-            if not (isinstance(self.maxval, (int, long)) and self.maxval > 0):
-                raise RuntimeError('Could not determine maxval from iterable. '
-                                   'You must explicitly provide a maxval.')
-        self._iterable = iter(iterable)
-        self.start()
-        return self
-
-    def __iter__(self):
-        return self
-
-    def next(self):
-        try:
-            next = self._iterable.next()
-            self.update(self.currval + 1)
-            return next
-        except StopIteration:
-            self.finish()
-            raise
-
-    def _handle_resize(self, signum, frame):
-        h, w = array('h', ioctl(self.fd, termios.TIOCGWINSZ, '\0' * 8))[:2]
-        self.term_width = w
-
-    def percentage(self):
-        "Returns the percentage of the progress."
-        return self.currval * 100.0 / self.maxval
-
-    def _format_widgets(self):
-        r = []
-        hfill_inds = []
-        num_hfill = 0
-        currwidth = 0
-        for i, w in enumerate(self.widgets):
-            if isinstance(w, ProgressBarWidgetHFill):
-                r.append(w)
-                hfill_inds.append(i)
-                num_hfill += 1
-            elif isinstance(w, basestring):
-                r.append(w)
-                currwidth += len(w)
-            else:
-                weval = w.update(self)
-                currwidth += len(weval)
-                r.append(weval)
-        for iw in hfill_inds:
-            widget_width = int((self.term_width - currwidth) // num_hfill)
-            r[iw] = r[iw].update(self, widget_width)
-        return r
-
-    def _format_line(self):
-        return ''.join(self._format_widgets()).ljust(self.term_width)
-
-    def _next_update(self):
-        return int((int(self.num_intervals *
-                        (self.currval / self.maxval)) + 1) *
-                   self.update_interval)
-
-    def _need_update(self):
-        """Returns true when the progressbar should print an updated line.
-
-        You can override this method if you want finer grained control over
-        updates.
-
-        The current implementation is optimized to be as fast as possible and
-        as economical as possible in the number of updates. However, depending
-        on your usage you may want to do more updates. For instance, if your
-        progressbar stays in the same percentage for a long time, and you want
-        to update other widgets, like ETA, then you could return True after
-        some time has passed with no updates.
-
-        Ideally you could call self._format_line() and see if it's different
-        from the previous _format_line() call, but calling _format_line() takes
-        around 20 times more time than calling this implementation of
-        _need_update().
-        """
-        return self.currval >= self.next_update
-
-    def update(self, value):
-        "Updates the progress bar to a new value."
-        assert 0 <= value <= self.maxval, '0 <= %d <= %d' % (value, self.maxval)
-        self.currval = value
-        if not self._need_update():
-            return
-        if self.start_time is None:
-            raise RuntimeError('You must call start() before calling update()')
-        now = time.time()
-        self.seconds_elapsed = now - self.start_time
-        self.next_update = self._next_update()
-        self.fd.write(self._format_line() + '\r')
-        self.last_update_time = now
-
-    def start(self):
-        """Starts measuring time, and prints the bar at 0%.
-
-        It returns self so you can use it like this:
-        >>> pbar = ProgressBar().start()
-        >>> for i in xrange(100):
-        ...    # do something
-        ...    pbar.update(i+1)
-        ...
-        >>> pbar.finish()
-        """
-        if self.maxval is None:
-            self.maxval = self._DEFAULT_MAXVAL
-        assert self.maxval > 0
-
-        self.num_intervals = max(100, self.term_width)
-        self.update_interval = self.maxval / self.num_intervals
-        self.next_update = 0
-
-        self.start_time = self.last_update_time = time.time()
-        self.update(0)
-        return self
-
-    def finish(self):
-        """Used to tell the progress is finished."""
-        self.finished = True
-        self.update(self.maxval)
-        self.fd.write('\n')
-        if self.signal_set:
-            signal.signal(signal.SIGWINCH, signal.SIG_DFL)

bitbake/lib/prserv/__init__.py

@@ -1,11 +0,0 @@
-__version__ = "1.0.0"
-
-import os, time
-import sys,logging
-
-def init_logger(logfile, loglevel):
-    numeric_level = getattr(logging, loglevel.upper(), None)
-    if not isinstance(numeric_level, int):
-        raise ValueError('Invalid log level: %s' % loglevel)
-    logging.basicConfig(level=numeric_level, filename=logfile)
-

bitbake/lib/prserv/db.py

@@ -1,100 +0,0 @@
-import logging
-import os.path
-import errno
-import sys
-import warnings
-import sqlite3
-
-try:
-    import sqlite3
-except ImportError:
-    from pysqlite2 import dbapi2 as sqlite3
-
-sqlversion = sqlite3.sqlite_version_info
-if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
-    raise Exception("sqlite3 version 3.3.0 or later is required.")
-
-class NotFoundError(StandardError):
-    pass
-
-class PRTable():
-    def __init__(self,cursor,table):
-        self.cursor = cursor
-        self.table = table
-
-        #create the table
-        self._execute("CREATE TABLE IF NOT EXISTS %s \
-                    (version TEXT NOT NULL, \
-                    checksum TEXT NOT NULL, \
-                    value INTEGER, \
-                    PRIMARY KEY (version,checksum));"
-                      % table)
-
-    def _execute(self, *query):
-        """Execute a query, waiting to acquire a lock if necessary"""
-        count = 0
-        while True:
-            try:
-                return self.cursor.execute(*query)
-            except sqlite3.OperationalError as exc:
-                if 'database is locked' in str(exc) and count < 500:
-                    count = count + 1
-                    continue
-                raise
-            except sqlite3.IntegrityError as exc:
-                print "Integrity error %s" % str(exc)
-                break
-
-    def getValue(self, version, checksum):
-        data=self._execute("SELECT value FROM %s WHERE version=? AND checksum=?;" % self.table,
-                           (version,checksum))
-        row=data.fetchone()
-        if row != None:
-            return row[0]
-        else:
-            #no value found, try to insert
-            self._execute("INSERT INTO %s VALUES (?, ?, (select ifnull(max(value)+1,0) from %s where version=?));" 
-                           % (self.table,self.table),
-                           (version,checksum,version))
-            data=self._execute("SELECT value FROM %s WHERE version=? AND checksum=?;" % self.table,
-                               (version,checksum))
-            row=data.fetchone()
-            if row != None:
-                return row[0]
-            else:
-                raise NotFoundError
-
-class PRData(object):
-    """Object representing the PR database"""
-    def __init__(self, filename):
-        self.filename=os.path.abspath(filename)
-        #build directory hierarchy
-        try:
-            os.makedirs(os.path.dirname(self.filename))
-        except OSError as e:
-            if e.errno != errno.EEXIST:
-                raise e
-        self.connection=sqlite3.connect(self.filename, timeout=5,
-                                          isolation_level=None)
-        self.cursor=self.connection.cursor()
-        self._tables={}
-
-    def __del__(self):
-        print "PRData: closing DB %s" % self.filename
-        self.connection.close()
-
-    def __getitem__(self,tblname):
-        if not isinstance(tblname, basestring):
-            raise TypeError("tblname argument must be a string, not '%s'" %
-                            type(tblname))
-        if tblname in self._tables:
-            return self._tables[tblname]
-        else:
-            tableobj = self._tables[tblname] = PRTable(self.cursor, tblname)
-            return tableobj
-
-    def __delitem__(self, tblname):
-        if tblname in self._tables:
-            del self._tables[tblname]
-        logging.info("drop table %s" % (tblname))
-        self.cursor.execute("DROP TABLE IF EXISTS %s;" % tblname) 

bitbake/lib/prserv/serv.py

@@ -1,198 +0,0 @@
-import os,sys,logging
-import signal,time, atexit
-from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
-import xmlrpclib,sqlite3
-
-import bb.server.xmlrpc
-import prserv
-import prserv.db
-
-if sys.hexversion < 0x020600F0:
-    print("Sorry, python 2.6 or later is required.")
-    sys.exit(1)
-
-class Handler(SimpleXMLRPCRequestHandler):
-    def _dispatch(self,method,params):
-        try:
-            value=self.server.funcs[method](*params)
-        except:
-            import traceback
-            traceback.print_exc()
-            raise
-        return value
-
-class PRServer(SimpleXMLRPCServer):
-    pidfile="/tmp/PRServer.pid"
-    def __init__(self, dbfile, logfile, interface, daemon=True):
-        ''' constructor '''
-        SimpleXMLRPCServer.__init__(self, interface,
-                                    requestHandler=SimpleXMLRPCRequestHandler,
-                                    logRequests=False, allow_none=True)
-        self.dbfile=dbfile
-        self.daemon=daemon
-        self.logfile=logfile
-        self.host, self.port = self.socket.getsockname()
-        self.db=prserv.db.PRData(dbfile)
-        self.table=self.db["PRMAIN"]
-
-        self.register_function(self.getPR, "getPR")
-        self.register_function(self.quit, "quit")
-        self.register_function(self.ping, "ping")
-        self.register_introspection_functions()
-
-    def ping(self):
-        return not self.quit
- 
-    def getPR(self, version, checksum):
-        try:
-            return self.table.getValue(version,checksum)
-        except prserv.NotFoundError:
-            logging.error("can not find value for (%s, %s)",version,checksum)
-            return None
-        except sqlite3.Error as exc:
-            logging.error(str(exc))
-            return None
-
-    def quit(self):
-        self.quit=True
-        return
-
-    def _serve_forever(self):
-        self.quit = False
-        self.timeout = 0.5
-        while not self.quit:
-            self.handle_request()
-
-        logging.info("PRServer: stopping...")
-        self.server_close()
-        return
-
-    def start(self):
-        if self.daemon is True:
-            logging.info("PRServer: starting daemon...")
-            self.daemonize()
-        else:
-            logging.info("PRServer: starting...")
-            self._serve_forever()
-
-    def delpid(self):
-        os.remove(PRServer.pidfile)
-
-    def daemonize(self):
-        """
-        See Advanced Programming in the UNIX, Sec 13.3
-        """
-        os.umask(0)
-
-        try:
-            pid = os.fork()
-            if pid > 0: 
-                sys.exit(0)
-        except OSError as e:
-            sys.stderr.write("1st fork failed: %d %s\n" % (e.errno, e.strerror))
-            sys.exit(1)
-
-        os.setsid()
-        """
-        fork again to make sure the daemon is not session leader, 
-        which prevents it from acquiring controlling terminal
-        """
-        try:
-            pid = os.fork()
-            if pid > 0: #parent
-                sys.exit(0)
-        except OSError as e:
-            sys.stderr.write("2nd fork failed: %d %s\n" % (e.errno, e.strerror))
-            sys.exit(1)
-
-        os.chdir("/")
-
-        sys.stdout.flush()
-        sys.stderr.flush()
-        si = file('/dev/null', 'r')
-        so = file(self.logfile, 'a+')
-        se = so
-        os.dup2(si.fileno(),sys.stdin.fileno())
-        os.dup2(so.fileno(),sys.stdout.fileno())
-        os.dup2(se.fileno(),sys.stderr.fileno())
-
-        # write pidfile
-        atexit.register(self.delpid)
-        pid = str(os.getpid()) 
-        pf = file(PRServer.pidfile, 'w+')
-        pf.write("%s\n" % pid)
-        pf.write("%s\n" % self.host)
-        pf.write("%s\n" % self.port)
-        pf.close()
-
-        self._serve_forever()
-
-class PRServerConnection():
-    def __init__(self, host, port):
-        self.connection = bb.server.xmlrpc._create_server(host, port)
-        self.host = host
-        self.port = port
-
-    def terminate(self):
-        # Don't wait for server indefinitely
-        import socket
-        socket.setdefaulttimeout(2)
-        try:
-            self.connection.quit()
-        except:
-            pass
-
-    def getPR(self, version, checksum):
-        return self.connection.getPR(version, checksum)
-
-    def ping(self):
-        return self.connection.ping()
-
-def start_daemon(options):
-    try:
-        pf = file(PRServer.pidfile,'r')
-        pid = int(pf.readline().strip())
-        pf.close()
-    except IOError:
-        pid = None
-
-    if pid:
-        sys.stderr.write("pidfile %s already exist. Daemon already running?\n"
-                            % PRServer.pidfile)
-        sys.exit(1)
-
-    server = PRServer(options.dbfile, interface=(options.host, options.port),
-                      logfile=os.path.abspath(options.logfile))
-    server.start()
-
-def stop_daemon():
-    try:
-        pf = file(PRServer.pidfile,'r')
-        pid = int(pf.readline().strip())
-        host = pf.readline().strip()
-        port = int(pf.readline().strip())
-        pf.close()
-    except IOError:
-        pid = None
-
-    if not pid:
-        sys.stderr.write("pidfile %s does not exist. Daemon not running?\n"
-                        % PRServer.pidfile)
-        sys.exit(1)
-
-    PRServerConnection(host,port).terminate()
-    time.sleep(0.5)
-
-    try:
-        while 1:
-            os.kill(pid,signal.SIGTERM)
-            time.sleep(0.1)
-    except OSError as err:
-        err = str(err)
-        if err.find("No such process") > 0:
-            if os.path.exists(PRServer.pidfile):
-                os.remove(PRServer.pidfile)
-        else:
-            print err
-            sys.exit(1)
-

bitbake/lib/pysh/builtin.py

@@ -151,7 +151,7 @@ def builtin_trap(name, args, interp, env, stdin, stdout, stderr, debugflags):
     for sig in args[1:]:
         try:
             env.traps[sig] = action
-        except Exception as e:
+        except Exception, e:
             stderr.write('trap: %s\n' % str(e))
     return 0
 
@@ -214,7 +214,7 @@ def utility_cat(name, args, interp, env, stdin, stdout, stderr, debugflags):
                     data = f.read()
                 finally:
                     f.close()
-            except IOError as e:
+            except IOError, e:
                 if e.errno != errno.ENOENT:
                     raise
                 status = 1
@@ -433,7 +433,7 @@ def utility_mkdir(name, args, interp, env, stdin, stdout, stderr, debugflags):
         if option.has_p:
             try:
                 os.makedirs(path)
-            except IOError as e:
+            except IOError, e:
                 if e.errno != errno.EEXIST:
                     raise
         else:               
@@ -561,7 +561,7 @@ def utility_sort(name, args, interp, env, stdin, stdout, stderr, debugflags):
                     lines = f.readlines()
                 finally:
                     f.close()
-            except IOError as e:
+            except IOError, e:
                 stderr.write(str(e) + '\n')
                 return 1
         
@@ -679,7 +679,7 @@ def run_command(name, args, interp, env, stdin, stdout,
             p = subprocess.Popen([name] + args, cwd=env['PWD'], env=exec_env, 
                     stdin=stdin, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
         out, err = p.communicate()
-    except WindowsError as e:
+    except WindowsError, e:
         raise UtilityError(str(e))
 
     if not unixoutput:

bitbake/lib/pysh/interp.py

@@ -248,7 +248,7 @@ class Redirections:
                     raise NotImplementedError('cannot open absolute path %s' % repr(filename))
             else:
                 f = file(filename, mode+'b')
-        except IOError as e:
+        except IOError, e:
             raise RedirectionError(str(e))
             
         wrapper = None
@@ -368,7 +368,7 @@ def resolve_shebang(path, ignoreshell=False):
         if arg is None:
             return [cmd, win32_to_unix_path(path)]
         return [cmd, arg, win32_to_unix_path(path)]
-    except IOError as e:
+    except IOError, e:
         if  e.errno!=errno.ENOENT and \
             (e.errno!=errno.EPERM and not os.path.isdir(path)): # Opening a directory raises EPERM
             raise
@@ -747,7 +747,7 @@ class Interpreter:
             for cmd in cmds:
                 try:
                     status = self.execute(cmd)
-                except ExitSignal as e:
+                except ExitSignal, e:
                     if sourced:
                         raise
                     status = int(e.args[0])
@@ -758,13 +758,13 @@ class Interpreter:
                 if 'debug-utility' in self._debugflags or 'debug-cmd' in self._debugflags:
                     self.log('returncode ' + str(status)+ '\n')
             return status
-        except CommandNotFound as e:
+        except CommandNotFound, e:
             print >>self._redirs.stderr, str(e)
             self._redirs.stderr.flush()
             # Command not found by non-interactive shell
             # return 127
             raise
-        except RedirectionError as e:
+        except RedirectionError, e:
             # TODO: should be handled depending on the utility status
             print >>self._redirs.stderr, str(e)
             self._redirs.stderr.flush()
@@ -948,7 +948,7 @@ class Interpreter:
                     status = self.execute(func, redirs)
                 finally:
                     redirs.close()
-            except ReturnSignal as e:
+            except ReturnSignal, e:
                 status = int(e.args[0])
                 env['?'] = status
             return status
@@ -1044,7 +1044,7 @@ class Interpreter:
                 
         except ReturnSignal:
             raise
-        except ShellError as e:
+        except ShellError, e:
             if is_special or isinstance(e, (ExitSignal,
                                             ShellSyntaxError, ExpansionError)):
                 raise e

bitbake/lib/pysh/pyshyacc.py

@@ -7,7 +7,6 @@
 
 """PLY grammar file.
 """
-import os.path
 import sys
 
 import pyshlex
@@ -649,10 +648,7 @@ def p_error(p):
 try:
     import pyshtables
 except ImportError:
-    outputdir = os.path.dirname(__file__)
-    if not os.access(outputdir, os.W_OK):
-        outputdir = ''
-    yacc.yacc(tabmodule = 'pyshtables', outputdir = outputdir, debug = 0)
+    yacc.yacc(tabmodule = 'pyshtables')
 else:
     yacc.yacc(tabmodule = 'pysh.pyshtables', write_tables = 0, debug = 0)
 
@@ -708,9 +704,6 @@ def format_commands(v):
         if v.reverse_status:
             name = '!' + name
         return [name, format_commands(v.commands)]
-    elif isinstance(v, Case):
-        name = ['Case']
-        name += [v.name, format_commands(v.items)]
     elif isinstance(v, SimpleCommand):
         name = ['SimpleCommand']
         if v.words: