There’s a classic bootstrapping paradox in provisioning. To install an OS, you need something already running on that machine. Historically, it was an image on CD or USB, which scales badly already if you have more than two or three machines. Moreover, what if you don’t have physical access to them?

PXE (a.k.a. Preboot Execution Environment) solves it by pushing bootstrap data into the network card’s firmware. That is right: before any disk is touched, before any OS exists, the machine’s NIC itself can pull an executable off the network and run it. The machine arrives with nothing and leaves with an OS.

This is why netboot basically underpins all bare-metal automation: datacenter provisioning, lab imaging, diskless workstations, Kubernetes node installation, every “reinstall this box” button which saves your day sometimes when you screw things up. Recently, I used our internal PXE to provision a newcomer’s PC because I did not find any spare USB drive.

What’s under the hood

The initial installation sequence is simpler than its reputation:

[NIC firmware] --DHCP DISCOVER (+ option 93: "I am arch X")--> [DHCP server] [NIC firmware] <--OFFER (+ option 67: "Boot this file")------ [DHCP server] [NIC firmware] --fetch bootfile-------------------------------> [file source] [bootloader] --fetch kernel + initramfs---------------------> [file source] [kernel] Let's boot
The interesting part here is that DHCP is doing double duty. Beyond handing out an IP, it also answers “What should I boot?” with a few extra options:

  • Option 93- client architecture provided by NIC.- 0means legacy BIOS x86,- 7and- 9mean x64 UEFI over TFTP, and- 16means x64 UEFI capable of HTTP boot. The last value makes very lightweight PXE setups possible.
  • Option 67- bootfile name. For earlier versions, it’s classically a filename like- pxelinux.0available on a file server. In case of- 16, it can be a full URL.

Historically, the file source was always TFTP, and this solution is not the best one: UDP, no encryption, no authentication, and a lockstep ack-every-block design which makes transferring a 40 Mb initramfs feel like a punishment. It was chosen because it fits in a ROM and is used today because of its backwards compatibility.

A lot of PXE tutorials actually concentrate on setting up a TFTP server, which is technically not complex but very suboptimal. Mainly because of newer firmware (number 16 above) that makes all of this obsolete: remember about HTTP boot capability?

Is it all about S3?

Once you notice that option 67 can hold http://, the architecture falls out immediately. UEFI firmware from roughly 2015 and onwards implements HTTP boot: they are capable of fetching an EFI binary over TCP and executing it. So you don’t need TFTP setup or any other file server at all, just something answering GET requests - for instance, an S3 bucket.

The resulting design can be very lightweight:

  • S3 holds every artifact: iPXE binary, boot script, kernel, initramfs.
  • DHCP holds one static URL.

And nothing else. We still use iPXE as an intermediate stage because raw UEFI HTTP Boot can only fetch and run a single EFI application: it has no scripting, no way to pass a kernel command line, no retries. iPXE gives us all of that - plus an opportunity to create an interactive menu - and it fits in about a megabyte.

There's one handshake detail worth knowing before you configure anything. An HTTP-boot client doesn't announce itself as a generic PXE client. It should send option 60 (vendor class) as something like HTTPClient:Arch:00016:UNDI:003016, and it expects the server to echo HTTPClient back in the reply. If your DHCP server (or proxy) doesn't, the client ignores the offer silently and without retries. This is probably the most common reason a correct-looking HTTP boot setup does nothing at all. A bit more detail is covered below under Step 3.

Let’s check how it works with Alpine Linux.

Step 1: Alpine in a bucket

```
BUCKET=my-netboot
REGION=eu-west-2
VER=v3.24

aws s3 mb "s3://$BUCKET" --region "$REGION"
curl -O "https://dl-cdn.alpinelinux.org/alpine/$VER/releases/x86_64/netboot/vmlinuz-lts"
curl -O "https://dl-cdn.alpinelinux.org/alpine/$VER/releases/x86_64/netboot/initramfs-lts"
curl -O "https://dl-cdn.alpinelinux.org/alpine/$VER/releases/x86_64/netboot/modloop-lts"
aws s3 cp vmlinuz-lts "s3://$BUCKET/alpine/"
aws s3 cp initramfs-lts "s3://$BUCKET/alpine/"
aws s3 cp modloop-lts "s3://$BUCKET/alpine/"
```

That’s all, three files: kernel, initramfs, and modloop. The latter is specific for the Alpine realm - a squashfs of kernel modules that Alpine mounts over HTTP after kernel is up, to keep initramfs minimal.

You don’t need S3 static website hosting. The plain REST endpoint already serves objects over HTTP and will perfectly do the trick, e.g.

http://my-netboot.s3.eu-west-2.amazonaws.com/alpine/vmlinuz-lts
Don’t forget about bucket policies: you always can restrict access via aws:SourceIp to your provisioning network’s egress addresses.

Step 2: iPXE binary

This is the only artifact you have to compile, and its entire job is to bootstrap the client into S3. Also, check our subsequent steps if you run this lab on a bare-metal provider as it may even be redundant.

To make things easier, you can conveniently use the official iPXE repo:

```
git clone https://github.com/ipxe/ipxe.git && cd ipxe/src
cat > embed.ipxe <<'EOF'

!ipxe

dhcp
chain http://my-netboot.s3.eu-west-2.amazonaws.com/boot.ipxe
EOF
make bin-x86_64-efi/ipxe.efi EMBED=embed.ipxe
aws s3 cp bin-x86_64-efi/ipxe.efi "s3://$BUCKET/"
`` The part in the middle is your embedded script. Without it, iPXE comes up and boots iPXE forever as DHCP has no other target by default. So just point to your S3 where it will findboot.ipxe` script which will do the rest of the lifting.

Step 3: Your DHCP

A machine that is netbooting has no IP address yet. It cannot route, and its DHCP request goes out as a layer-2 broadcast which won’t cross routers. So whatever answers have to be sitting within the same network segment as the machine, or the machine should have an IP and a URL so it can route further to the internet.

You don’t need to configure anything around DHCP for testing on your PC: most UEFI firmware lets you add an HTTP boot entry manually in the boot menu, so type the S3 URL of ipxe.efi. Also, the majority of bare-metal providers already have this part covered and just allow you to enter a boot script completely bypassing the previous step.

But this is the only piece that cannot live in S3, so it’s worth understanding before we move forward. And here’s nothing new: something in your network already assigns addresses, so we should put the familiar configuration.

Match the clients where option 93 (client architecture) equals 16, and send them:

  • Option 67 (boot filename) with URL to ipxe.efiin S3;
  • Option 60 (vendor class) as HTTPClient.

If you can’t touch your primary DHCP, you can proxy these parameters via dnsmasq: run it on any Linux machine sitting in the same VLAN as the client. It will need UDP ports 67 and 4011 and should not be behind NAT.

Here’s a rough dnsmasq config which allows the proxy to return the parameters above and the ipxe.efi URL:

```

/etc/dnsmasq.d/netboot.conf

port=0 # no DNS, only boot info
dhcp-range=192.168.1.0,proxy # no address leases
dhcp-match=set:efi-http,option:client-arch,16
dhcp-userclass=set:ipxe,iPXE
dhcp-boot=tag:efi-http,tag:!ipxe,http://my-netboot.s3.eu-west-2.amazonaws.com/ipxe.efi
dhcp-option-force=tag:efi-http,tag:!ipxe,60,HTTPClient
```

Note that it is configured to return boot info only and does not provide address leases, so it does not conflict with the actual network’s DHCP server. Still, this approach might have some constraints which are mentioned below.

Step 4: Boot script

This is the target file having all the kernel parameters. Upload it to S3 as boot.ipxe:

```

!ipxe

set base http://my-netboot.s3.eu-west-2.amazonaws.com
kernel ${base}/alpine/vmlinuz-lts \
initrd=initramfs-lts \
modloop=${base}/alpine/modloop-lts \
alpine_repo=https://dl-cdn.alpinelinux.org/alpine/v3.24/main \
modules=loop,squashfs,sd-mod,usb-storage \
ip=dhcp console=tty0 quiet
initrd ${base}/alpine/initramfs-lts
boot
`` Here,modloopandalpine_repomake this scenario into a usable OS rather than a rescue prompt: you are already familiar with modloop from above, and the repository URL gives you a working package manager.alpine_repo` also can be your own S3 mirror, by the way.

You can also test this without touching hardware by running QEMU with OVMF, a UEFI firmware build from the EDK2 project, which gives you a real HTTP Boot implementation. I haven't run this path myself, so treat it as a direction rather than a recipe.

The things that will bite

At that point, you will have a working S3 setup for OS provisioning. As a bonus, it is configurable and does not have any TFTP or webserver machinery. If you want to change what your fleet boots, just upload other files.

But this text would not be complete without some warnings.

First of all, mind S3 egress. Every single boot pulls the full kernel, initramfs, and modloop, a few hundred megabytes on x86_64. At AWS’ prices, one machine is free and a 200-node reimage is a rounding error, but a CI fleet that reboots continuously is not. Put CloudFront in front of the bucket, or run a caching proxy at each site.

Secondly, this is plaintext HTTP. So anyone on the way can see and modify what you boot. To counter it, you can enable HTTPS which is available with iPXE, but not by default: see src/config/general.h for details.

Also, be aware of Secure Boot: if you test on a recent laptop or some Dell server, you might get a very laconic Exec format error because the iPXE binary you compiled is not signed by Microsoft. This can be cured by your own keys enrolled in the firmware - good for production.

Then, there is DHCP snooping. If your dnsmasq proxy works on your home network and does nothing at all at the office, this is probably why. Managed switches can be configured to drop DHCP server replies arriving on untrusted ports: it’s a standard defence against rogue DHCP servers, and from the switch's point of view your proxy is exactly that. As a result, dnsmasq receives the DISCOVER, logs that it matched your tag and sent the boot info, and tcpdump on the proxy shows the reply leaving. The client simply never gets it. In this case, you have to configure the primary DHCP server.

And, lastly, firmware quality is different. Some UEFI implementations don’t work well with DHCP proxies, others require some manual parameters to be set, so test on actual hardware anyway.