Mounted 50MB ext2 image file with loop
First I added busybox and glibc6, bash, libncurses, sysvinit, mount, libuuid, libselinux1, libsepol1, mktemp, libacl1, libattr1, libpam-modules, libpam0g, libpam-runtime, libdb4, libcap1, gcc-4.1-base, login_4, passwd, lsb-base, makedev,
Trying something new:
* Installing coreutils, libc6, bash, perl-base, and dpkg manually, then you have to trick it into thinking that tzdata is actually installed, so that dpkg can then start its own database of installed packages, and get libc6 installed, as almost everything else is dependent upon it.
Now using a script to wget and install debs, then need to register that they are installed and the files that they install into var/lib/dpkg/info/packagename.list.
The goal here is to simply aid in the creation of small embedded debian systems. These systems will be "complete" once they are built, and therefore do not have a need for ongoing maintenance. Given the power and convenience of dpkg, it makes sense to chroot into an environment and install packages there. To do so, dpkg needs some help getting going.
So... run test.sh from debian host within target directory, then add fake tzdata to var/lib/dpkg/status and var/lib/dpkg/info/tzdata.list, then chroot into target directory and run dpkg.sh. ?
Additional tasks:
* Populate dev/
* mkdir proc
* clean locale, man, doc, linda, lintian,
* remove ./lib/tls/libnss_nis*, ./lib/tls/libnss_hesiod*, ./lib/libnss_nishttp://wiki.xensource.com/xenwiki/InstallGuestImage
I'm still not done yet but I'm finally happy with how this is working out. I knew there was a way to make a relatively healthy debian linux installation using only debian packages that would be really really small. I tried forcing debootstrap to minimize itself, but that didn't work, then I tried mixing and matching stuff from ttylinux and debian, and that kind of worked but also didn't kind of work.
Now I'm using a shell script to manually install a bunch of debian packages into a target directory and start a basic dpkg database even with some fake data (for tzdata), then chroot to that target directory and install those same deb packages using dpkg. It actually works, and you end up with a chroot environment much smaller than the one created by debootstrap. The one I have just finished building is only 28M, and I am using squashfs to compress it down, where it gets to be 11M!
That includes the following debian packages:
Which is surprisingly enough to boot the system with a corresponding kernel. I'm going to add more stuff, perhaps udev, perhaps incron. I did wipe out /usr/share/locale, /usr/share/doc, and /usr/share/man, too.
I chose squashfs because I do plan to add a lot more stuff, and squashfs is helpful when it comes to conserving ram. I also like the fact that I am able to use a ramfs overlay for stuff like logs and bash history files. That is pretty cool.
Right now the scripts I used to build the target are very dumb, but I had to make it work before I sat down and planned something more sophisticated out, as I really wasn't sure if it was possible. Now that I know its possible, I'll use phing to make the process more elegant and modular.
In a nutshell:
* Create file-based disk image with dd
* Create partition table
* Setup disk and partition using losetup
* Format partition
* Install grub
* Add kernel, system map, initrd.gz, and final.sqfs to partition
* Unmount partition and disk image
* Copy to media - compact flash card, etc.
The initrd I'm using right now is 8MB, though I'm sure I can reduce that down to 2 or 3MB. As it is, it compresses down to 1.1MB at the best compression ratio gzip has to offer. I'm not sure if or how I can unmount the initrd after the pivot_root in linuxrc. I hope so! (? ) Actually I don't think it can be unmounted, because it is used to mount the media which contains the squashfs filesystem. To conserve ram, I shrunk the initrd down to less than 2MB. I could probably go smaller with a smaller busybox, but I think that 2MB is OK for now. My guess is that to get it smaller, I'd need to use uclibc and a corresponding busybox. That might be feasible and worth it if I have the time later on. (Did a quick check at the debian site - their busybox static is 1.5M, and the busybox-udeb requires uclibc which adds up to more, so maybe the initrd I have now is the best I can get!)
This process is used by m0n0wall and the m0n0dev.php scripts which inspired the first nodows phing buildfile, but I had no idea how to do anything like that with linux. It took me until ttylinux to get a grasp for the minimal requirements to boot and run linux. Then, with the help of Damn Small Linux and the knoppix cloop concepts, I found a page describing how to boot a squashfs filesystem using initrd, pivot_root, and busybox. Some of that overlapped with what I had learned from ttylinux, so it all came together!
There is still a lot of work to do on these debian scripts, but this is a great step forward. Hooray! :-)
Requirements:
* Squashfs + Unionfs capable kernel
* ext2 disk image for storing /boot/ and squashfs filesystem
* initrd with busybox for mounting squashfs and unionfs
* final squashed filesystem
* Can you save stuff back to the top level partition for persistence? Yes, next step is to fire up php and parse some XML!
* Question: does it make sense to use a daemon to listen for inotify events (like incron), so that if and when a file is changed, the corresponding service is reloaded? For example, XML file contains interface configuration, so it would output to /etc/network/interfaces, when that file is changed, "/etc/init.d/networking restart" could be called. That way, init can do its thing, and as its going through the runlevels, the php process would be called and that would output new files for what it is configuring, such as hostname, localtime, /etc/apache2/apache2.conf, etc. As the system is running, changes made to the xml document would cause the php parsing process to restart, and the process would begin again.
Basic build scripts can be found here.
Unionfs info: http://www.linuxjournal.com/article/7714http://www.fsl.cs.sunysb.edu/docs/sipek-ols2006/index.htmlhttp://penguin-soft.com/penguin/man/4/unionfs.html
First I added busybox and glibc6, bash, libncurses, sysvinit, mount, libuuid, libselinux1, libsepol1, mktemp, libacl1, libattr1, libpam-modules, libpam0g, libpam-runtime, libdb4, libcap1, gcc-4.1-base, login_4, passwd, lsb-base, makedev,
Trying something new:
* Installing coreutils, libc6, bash, perl-base, and dpkg manually, then you have to trick it into thinking that tzdata is actually installed, so that dpkg can then start its own database of installed packages, and get libc6 installed, as almost everything else is dependent upon it.
Now using a script to wget and install debs, then need to register that they are installed and the files that they install into var/lib/dpkg/info/packagename.list.
The goal here is to simply aid in the creation of small embedded debian systems. These systems will be "complete" once they are built, and therefore do not have a need for ongoing maintenance. Given the power and convenience of dpkg, it makes sense to chroot into an environment and install packages there. To do so, dpkg needs some help getting going.
So... run test.sh from debian host within target directory, then add fake tzdata to var/lib/dpkg/status and var/lib/dpkg/info/tzdata.list, then chroot into target directory and run dpkg.sh. ?
Additional tasks:
* Populate dev/
* mkdir proc
* clean locale, man, doc, linda, lintian,
* remove ./lib/tls/libnss_nis*, ./lib/tls/libnss_hesiod*, ./lib/libnss_nishttp://wiki.xensource.com/xenwiki/InstallGuestImage
du -Sh | grep M
2.2M ./lib/tls
3.7M ./lib
2.5M ./bin
1.5M ./sbin
1.5M ./usr/lib/i686/cmov
1.4M ./usr/lib/i586
1.5M ./usr/lib/i486
2.6M ./usr/lib
5.4M ./usr/binI'm still not done yet but I'm finally happy with how this is working out. I knew there was a way to make a relatively healthy debian linux installation using only debian packages that would be really really small. I tried forcing debootstrap to minimize itself, but that didn't work, then I tried mixing and matching stuff from ttylinux and debian, and that kind of worked but also didn't kind of work.
Now I'm using a shell script to manually install a bunch of debian packages into a target directory and start a basic dpkg database even with some fake data (for tzdata), then chroot to that target directory and install those same deb packages using dpkg. It actually works, and you end up with a chroot environment much smaller than the one created by debootstrap. The one I have just finished building is only 28M, and I am using squashfs to compress it down, where it gets to be 11M!
-rwx------ 1 root root 11M 2007-10-16 15:37 ../debtty.sqfsThat includes the following debian packages:
# dpkg -l
Desired=Unknown/Install/Remove/Purge/Hold| Status=Not/Installed/Config-files/Unpacked/Failed-config/Half-installed|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)||/ Name Version Description+++-==============-==============-============================================ii base-files 4 Debian base system miscellaneous filesii base-passwd 3.5.11 Debian base system master password and groupii bash 3.1dfsg-8 The GNU Bourne Again SHellii coreutils 5.97-5.3 The GNU core utilitiesii debianutils 2.17 Miscellaneous utilities specific to Debianii dpkg 1.13.25 package maintenance system for Debianii e2fslibs 1.39+1.40-WIP- ext2 filesystem librariesii e2fsprogs 1.39+1.40-WIP- ext2 file system utilities and librariesii gawk 3.1.5.dfsg-4 GNU awk, a pattern scanning and processing lii gcc-4.1-base 4.1.1-21 The GNU Compiler Collection (base package)ii initscripts 2.86.ds1-38 Scripts for initializing and shutting down tii libacl1 2.2.41-1 Access control list shared libraryii libattr1 2.4.32-1 Extended attribute shared libraryii libblkid1 1.39+1.40-WIP- block device id libraryii libc6 2.3.6.ds1-13et GNU C Library: Shared librariesii libcap1 1.10-14 support for getting/setting POSIX.1e capabilii libcomerr2 1.39+1.40-WIP- common error description libraryii libdb4.3 4.3.29-8 Berkeley v4.3 Database Libraries [runtime]ii libdevmapper1. 1.02.08-1 The Linux Kernel Device Mapper userspace libii libgcc1 4.1.1-21 GCC support libraryii libncurses5 5.5-5 Shared libraries for terminal handlingii libpam-modules 0.79-4 Pluggable Authentication Modules for PAMii libpam-runtime 0.79-4 Runtime support for the PAM libraryii libpam0g 0.79-4 Pluggable Authentication Modules libraryii libselinux1 1.32-3 SELinux shared librariesii libsepol1 1.14-2 Security Enhanced Linux policy library for cii libslang2 2.0.6-4 The S-Lang programming library - runtime verii libss2 1.39+1.40-WIP- command-line interface parsing libraryii libuuid1 1.39+1.40-WIP- universally unique id libraryii login 4.0.18.1-7 system login toolsii lsb-base 3.1-23.2etch1 Linux Standard Base 3.1 init script functionii mktemp 1.5-2 Makes unique filenames for temporary filesii mount 2.12r-19 Tools for mounting and manipulating filesystii ncurses-bin 5.5-5 Terminal-related programs and man pagesii passwd 4.0.18.1-7 change and administer password and group datii perl-base 5.8.8-7 The Pathologically Eclectic Rubbish Listerii procps 3.2.7-3 /proc file system utilitiesii sed 4.1.5-1 The GNU sed stream editorii sysv-rc 2.86.ds1-38 System-V-like runlevel change mechanismii sysvinit-utils 2.86.ds1-38 System-V-like utilitiesii tzdata 2007b-1 Time Zone and Daylight Saving Time Dataii util-linux 2.12r-19 Miscellaneous system utilitiesii zlib1g 1.2.3-13 compression library - runtimeWhich is surprisingly enough to boot the system with a corresponding kernel. I'm going to add more stuff, perhaps udev, perhaps incron. I did wipe out /usr/share/locale, /usr/share/doc, and /usr/share/man, too.
I chose squashfs because I do plan to add a lot more stuff, and squashfs is helpful when it comes to conserving ram. I also like the fact that I am able to use a ramfs overlay for stuff like logs and bash history files. That is pretty cool.
Right now the scripts I used to build the target are very dumb, but I had to make it work before I sat down and planned something more sophisticated out, as I really wasn't sure if it was possible. Now that I know its possible, I'll use phing to make the process more elegant and modular.
In a nutshell:
* Create file-based disk image with dd
* Create partition table
* Setup disk and partition using losetup
* Format partition
* Install grub
* Add kernel, system map, initrd.gz, and final.sqfs to partition
* Unmount partition and disk image
* Copy to media - compact flash card, etc.
The initrd I'm using right now is 8MB, though I'm sure I can reduce that down to 2 or 3MB. As it is, it compresses down to 1.1MB at the best compression ratio gzip has to offer. I'm not sure if or how I can unmount the initrd after the pivot_root in linuxrc. I hope so! (? ) Actually I don't think it can be unmounted, because it is used to mount the media which contains the squashfs filesystem. To conserve ram, I shrunk the initrd down to less than 2MB. I could probably go smaller with a smaller busybox, but I think that 2MB is OK for now. My guess is that to get it smaller, I'd need to use uclibc and a corresponding busybox. That might be feasible and worth it if I have the time later on. (Did a quick check at the debian site - their busybox static is 1.5M, and the busybox-udeb requires uclibc which adds up to more, so maybe the initrd I have now is the best I can get!)
This process is used by m0n0wall and the m0n0dev.php scripts which inspired the first nodows phing buildfile, but I had no idea how to do anything like that with linux. It took me until ttylinux to get a grasp for the minimal requirements to boot and run linux. Then, with the help of Damn Small Linux and the knoppix cloop concepts, I found a page describing how to boot a squashfs filesystem using initrd, pivot_root, and busybox. Some of that overlapped with what I had learned from ttylinux, so it all came together!
There is still a lot of work to do on these debian scripts, but this is a great step forward. Hooray! :-)
Requirements:
* Squashfs + Unionfs capable kernel
* ext2 disk image for storing /boot/ and squashfs filesystem
* initrd with busybox for mounting squashfs and unionfs
* final squashed filesystem
* Can you save stuff back to the top level partition for persistence? Yes, next step is to fire up php and parse some XML!
* Question: does it make sense to use a daemon to listen for inotify events (like incron), so that if and when a file is changed, the corresponding service is reloaded? For example, XML file contains interface configuration, so it would output to /etc/network/interfaces, when that file is changed, "/etc/init.d/networking restart" could be called. That way, init can do its thing, and as its going through the runlevels, the php process would be called and that would output new files for what it is configuring, such as hostname, localtime, /etc/apache2/apache2.conf, etc. As the system is running, changes made to the xml document would cause the php parsing process to restart, and the process would begin again.
Basic build scripts can be found here.
Unionfs info: http://www.linuxjournal.com/article/7714http://www.fsl.cs.sunysb.edu/docs/sipek-ols2006/index.htmlhttp://penguin-soft.com/penguin/man/4/unionfs.html