How to use Icecream with Yocto Project® builds

Objectives

This post shows you how to use Icecream with Yocto Project® builds – ideally using the “Reliable Embedded Systems Yocto Project®/OpenEmbedded build framework“.

Prerequisites

This build framework uses a docker build container to run “BitBake“. The build framework already contains Icecream [more details here] as an icecc-container and in the build-container are the relevant components installed as well. If you want to learn more about the Yocto Project® and Icecream check out this video:

Icecream

The git repo of Icecream mentions: “Icecream was created by SUSE based on distcc. Like distcc, Icecream takes compile jobs from a build and distributes them among remote machines allowing a parallel build. But unlike distcc, Icecream uses a central server that dynamically schedules the compile jobs to the fastest free server. This advantage pays off mostly for shared computers, if you’re the only user on x machines, you have full control over them.”

Icecream-sundae

Icecream-sundae is a Commandline Monitor for Icecream written by our friend Joshua Watt.

Setup

Overview

We have three machines available for our little experiment. Let’s assume they are called e450-4, e450-5 and e450-6.

Machine	icecc-scheduler	iceccd	icecream-sundae	Bitbake build
`e450-4`	yes	yes	yes
`e450-5`		yes	optional	yes
`e450-6`		yes	optional

What should run on which machine?

e450-4

icecc-container

Let’s start the icecc-container with iceccd and the icecc-scheduler:

student@e450-4:/workdir⟫ 
cd /workdir/crops-container-x86-64/icecc-container/non-local_scripts
./docker_run.sh reliableembeddedsystems/icecream-container:2021-04-08 sched

...
++ docker run --name icecream_container -d -e ICECC_ENABLE_SCHEDULER=1 --net=host reliableembeddedsystems/icecream-container:2021-04-08
Unable to find image 'reliableembeddedsystems/icecream-container:2021-04-08' locally
2021-04-08: Pulling from reliableembeddedsystems/icecream-container
004f1eed87df: Pull complete 
4b53851c008a: Pull complete 
fcc89a6ff4b7: Pull complete 
4b147887ebe6: Pull complete 
579faa19e08b: Pull complete 
Digest: sha256:88ab1391ad5f997ee7d9877e9cb59b24e5d90ed079eeaed2cd05d280b9965950
Status: Downloaded newer image for reliableembeddedsystems/icecream-container:2021-04-08
765f6d502e0f4a30d2578102c72791199c0c652e294f3c3fe240db969fa5a342

icecream-sundae

Let’s start Icecream-sundae from the same or another shell on the same machine:

student@e450-4:~⟫ 
cd /workdir/crops-container-x86-64/icecc-container/non-local_scripts
./icecream-sundae.sh
Command line Icecream status monitor, Version 1.0.0
Copyright (C) 2018 by Garmin Ltd. or its subsidiaries.
This is free software, and you are welcome to redistribute it
under certain conditions; run with '--about' for details.
ignoring localhost lo
broadcast enp3s0 192.168.42.255
broadcast docker0 172.17.255.255
Suitable scheduler found at 192.168.42.104:8765 (version: 39)
Suitable scheduler found at 192.168.42.104:8765 (version: 39)
Selecting scheduler at 192.168.42.104:8765
scheduler is on 192.168.42.104:8765 (net ICECREAM)
Done waiting
Got scheduler 192.168.42.104

You should see something like that, which means that iceccd and the icecc-scheduler are running:

Scheduler: 192.168.42.104 Netname: ICECREAM
Servers: Total:1 Available:1 Active:0
Total: Remote:0 Local:0
Jobs: Maximum:8 Active:0 Local:0 Pending:0
      [        ]

v ID NAME   IN    CUR MAX JOBS       OUT   LOCAL ACTIVE PENDING SPEED
+ 1  e450-4 0     0   8   [        ] 0     0     0      0       0

e450-6

iceccd

Here we only need iceccd as explained in our plan from above.

student@e450-6:/workdir⟫ 
cd /workdir/crops-container-x86-64/icecc-container/non-local_scripts
./docker_run.sh reliableembeddedsystems/icecream-container:2021-04-08

Monitor changes with Icecream-sundae

Scheduler: 192.168.42.104 Netname: ICECREAM
Servers: Total:2 Available:2 Active:0
Total: Remote:0 Local:0
Jobs: Maximum:16 Active:0 Local:0 Pending:0
      [                ]

v ID NAME   IN    CUR MAX JOBS       OUT   LOCAL ACTIVE PENDING SPEED
+ 1  e450-4 0     0   8   [        ] 0     0     0      0       0
+ 2  e450-6 0     0   8   [        ] 0     0     0      0       0

e450-5

Our plan says that we want iceccd. Also BitBake will be started here in a build-container.

iceccd

That’s pretty much the same story with what you saw on the machine e450-6.

student@e450-5:/workdir⟫ 
cd /workdir/crops-container-x86-64/icecc-container/non-local_scripts
./docker_run.sh reliableembeddedsystems/icecream-container:2021-04-08

...
++ docker run --name icecream_container -d --net=host reliableembeddedsystems/icecream-container:2021-04-08
Unable to find image 'reliableembeddedsystems/icecream-container:2021-04-08' locally
2021-04-08: Pulling from reliableembeddedsystems/icecream-container
004f1eed87df: Pull complete 
4b53851c008a: Pull complete 
fcc89a6ff4b7: Pull complete 
4b147887ebe6: Pull complete 
579faa19e08b: Pull complete 
Digest: sha256:88ab1391ad5f997ee7d9877e9cb59b24e5d90ed079eeaed2cd05d280b9965950
Status: Downloaded newer image for reliableembeddedsystems/icecream-container:2021-04-08
d182381e640e6b601fbbfe05863af039c0a8f8b4bb602b6e24d080fae10f4755

Apparently, if you just want to add another builder to the network you could just execute the following on any machine even without installing the “Reliable Embedded Systems Yocto Project®/OpenEmbedded build framework“:

docker run --name icecream_container -d --net=host reliableembeddedsystems/icecream-container:2021-04-08

Monitor changes with Icecream-sundae

If everything works as expected, Icecream-sundae, which runs on the machine e450-4, should show that machine e450-5 has joined:

Scheduler: 192.168.42.104 Netname: ICECREAM
Servers: Total:3 Available:3 Active:0
Total: Remote:0 Local:0
Jobs: Maximum:24 Active:0 Local:0 Pending:0
      [                        ]

v ID NAME   IN    CUR MAX JOBS       OUT   LOCAL ACTIVE PENDING SPEED
+ 1  e450-4 0     0   8   [        ] 0     0     0      0       0
+ 2  e450-6 0     0   8   [        ] 0     0     0      0       0
+ 3  e450-5 0     0   8   [        ] 0     0     0      0       0

build container

Make sure that /workdir/resy-poky-container.sh is configured properly:

#!/bin/bash
...
USE_ICECC="yes"
...
if [[ $USE_ICECC = yes ]]; then
   ICECC="--net=host "
fi

Start up the build container:

student@e450-5:/workdir
./resy-poky-container.sh

pick build target/fetch sources/inspect some variables

pokyuser@e450-5:/workdir/build$ 
# let's pick some generic multi-v7 target:
source /workdir/resy-cooker.sh multi-v7-ml-master
# don't yet configure the icecc stuff, but let's fetch whatever is needed
bitbake core-image-minimal --runall=fetch
# let's try to determine what are the "default" values for some of the values we will change:
bitbake -e | grep ^PARALLEL_MAKE=
PARALLEL_MAKE="-j 8"
bitbake -e | grep ^BB_NUMBER_THREADS=
BB_NUMBER_THREADS="-j 8"

adjust local.conf

student@e450-5:~⟫ 
cd /workdir/build/multi-v7-ml-master/conf
vim local.conf

# --> icecc
# total number of local/builder CPU cores * 2 or * 3?
# This variable usually takes the form of “-j x”, 
# where x represents the maximum number of parallel threads make can run.
# needs some playing
# 68 CPU cores total in my setup
ICECC_PARALLEL_MAKE?= "-j ${@oe.utils.cpu_count()*3}"
######################################################################
# Problem:  Icecream and sstate can be combined, however inheriting
#           icecc.bbclass changes most taskhashes
#           Which makes the SSTATE created with icecc unusable for 
#           builds without it.
# Solution: Always inherit icecc.bbclass and use ICECC_DISABLED ?= “1”
#           to turn off icecream for builds without icecc which want to
#           use the SSTATE created with icecc.

INHERIT += "icecc"
ICECC_DISABLED ??= "0"
######################################################################

# Blacklist packages from compiling with icecream. There really needs to be a
# better way to share this list...
# let's start with an empty Blacklist and see how things are going
ICECC_USER_PACKAGE_BL += "glib-2.0 bazel-native tensorflow-native"
# <-- icecc

build

pokyuser@e450-5:/workdir/build$ bitbake core-image-minimal

Monitor situation with Icecream-sundae

You might want to do some more fine-tuning. Here you see that all three machines are busy:

Scheduler: 192.168.42.104 Netname: ICECREAM
Servers: Total:3 Available:3 Active:3
Total: Remote:2646 Local:1615   
Jobs: Maximum:24 Active:24 Local:1 Pending:3
      [%=======================]

v ID NAME   IN    CUR MAX JOBS       OUT   LOCAL ACTIVE PENDING SPEED
+ 1  e450-4 912   8   8   [========] 0     0     0      0       0
+ 2  e450-6 1522  8   8   [========] 0     0     0      0       0
+ 3  e450-5 212   8   8   [%=======] 2646  1615  24     3       83.6531

With a few more machines it might look like this:

Scheduler: 192.168.42.104 Netname: ICECREAM
Servers: Total:10 Available:10 Active:10
Total: Remote:11617 Local:5349   
Jobs: Maximum:68 Active:54 Local:0 Pending:0
      [======================================================              ]

v ID NAME    IN    CUR MAX JOBS       OUT   LOCAL ACTIVE PENDING SPEED
+ 3  e450-5  966   2   8   [==      ] 11617 5349  54     0       70.4266
+ 9  e450-12 41    2   4   [==  ]     0     0     0      0       0
+ 8  e450-11 67    4   4   [====]     0     0     0      0       47.3424
+ 10 e450-13 79    4   4   [====]     0     0     0      0       79.2168
+ 7  e450-10 111   6   8   [======  ] 0     0     0      0       40.9161
+ 1  e450-4  4243  7   8   [======= ] 0     0     0      0       101.404
+ 2  e450-6  4314  7   8   [======= ] 0     0     0      0       110.203
+ 4  e450-3  1451  7   8   [======= ] 0     0     0      0       158.049
+ 5  e450-7  173   7   8   [======= ] 0     0     0      0       0
+ 6  e450-8  172   8   8   [========] 0     0     0      0       47.6192

Benchmark

The Icecream cluster consists of ten machines (including the build machine) and a total of 68 CPUs. We don’t use any SSTATE_MIRRORS. Therefore it’s enough to clean only the local “sstate“. We will build core-image-minimal with and without the Icecream cluster and measure how long it takes with time(1).

Benchmark with Icecream cluster

Clean local sstate

rm -rf /workdir/build/multi-v7-ml-master/tmp/
rm -rf /workdir/sstate_master/

Adjust local.conf

It should be fine if you use the one from above. Make sure Icecream is enabled:

ICECC_DISABLED ?= "0"

Test run

Let’s monitor the Icecream cluster activity with Icecream-sundae which runs on the machine e450-4 in order to ensure that Icecream is actually being used.

pokyuser@e450-5:/workdir/build/multi-v7-ml-master$ time bitbake core-image-minimal

NOTE: Started PRServer with DBfile: /workdir/build/multi-v7-ml-master/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 43990, PID: 22009
Loading cache: 100% |                                                                                                                                                         | ETA:  --:--:--
Loaded 0 entries from dependency cache.
Parsing recipes: 100% |########################################################################################################################################################| Time: 0:01:22
Parsing of 2393 .bb files complete (0 cached, 2393 parsed). 3676 targets, 140 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.49.2"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "ubuntu-18.04"
TARGET_SYS           = "arm-resy-linux-gnueabi"
MACHINE              = "multi-v7-ml"
DISTRO               = "resy"
DISTRO_VERSION       = "3.3"
TUNE_FEATURES        = "arm armv7a vfp thumb callconvention-hard"
TARGET_FPU           = "hard"
meta
meta-poky
meta-yocto-bsp       = "master:e1839b58ebe05242a52fe050aa9a08140136aa0a"
meta-multi-v7-ml-bsp-master = "master:00f716ef03a6d37489cf8f1704a566938d3c7c64"
meta-resy-master     = "master:48562f43bed481ad342d92b9cdcb45fa75487f58"
meta-oe
meta-networking
meta-filesystems
meta-python          = "master:5d8dfa1ad4ac5d32536a10c9e299a3dd17296fc9"

Initialising tasks: 100% |#####################################################################################################################################################| Time: 0:00:02
Sstate summary: Wanted 973 Local 0 Network 0 Missed 973 Current 0 (0% match, 0% complete)
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 2628 tasks of which 0 didn't need to be rerun and all succeeded.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 5 seconds

real    69m45.366s
user    0m14.576s
sys     0m2.208s

Benchmark without Icecream cluster

Let’s clean the local SSTATE

rm -rf /workdir/build/multi-v7-ml-master/tmp/
rm -rf /workdir/sstate_master/

Adjust local.conf

Make sure Icecream is disabled:

ICECC_DISABLED ?= "1"

Test run

Let’s monitor the Icecream cluster activity with Icecream-sundae which runs on the machine e450-4 in order to ensure that Icecream is actually not being used.

pokyuser@e450-5:/workdir/build/multi-v7-ml-master$ time bitbake core-image-minimal

NOTE: Started PRServer with DBfile: /workdir/build/multi-v7-ml-master/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 33841, PID: 32237
Loading cache: 100% |                                                                                                                                                         | ETA:  --:--:--
Loaded 0 entries from dependency cache.
Parsing recipes: 100% |########################################################################################################################################################| Time: 0:01:25
Parsing of 2393 .bb files complete (0 cached, 2393 parsed). 3676 targets, 140 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.49.2"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "ubuntu-18.04"
TARGET_SYS           = "arm-resy-linux-gnueabi"
MACHINE              = "multi-v7-ml"
DISTRO               = "resy"
DISTRO_VERSION       = "3.3"
TUNE_FEATURES        = "arm armv7a vfp thumb callconvention-hard"
TARGET_FPU           = "hard"
meta                 
meta-poky            
meta-yocto-bsp       = "master:e1839b58ebe05242a52fe050aa9a08140136aa0a"
meta-multi-v7-ml-bsp-master = "master:00f716ef03a6d37489cf8f1704a566938d3c7c64"
meta-resy-master     = "master:48562f43bed481ad342d92b9cdcb45fa75487f58"
meta-oe              
meta-networking      
meta-filesystems     
meta-python          = "master:5d8dfa1ad4ac5d32536a10c9e299a3dd17296fc9"

Initialising tasks: 100% |#####################################################################################################################################################| Time: 0:00:02
Sstate summary: Wanted 973 Local 0 Network 0 Missed 973 Current 0 (0% match, 0% complete)
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 2628 tasks of which 0 didn't need to be rerun and all succeeded.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 3 seconds

real    88m8.270s
user    0m15.180s
sys     0m2.608s

Benchmark Result

Icecream	total time (min./sec.)	time (sec.)	increase (%)	decrease (%)
Icecream utilized	69m45.366s	4185.366s		20.86%
Icecream not used	88m8.270s	5288.270	26.35%

Conclusion

If we compare the build time without and with the Icecream cluster, we notice a decrease of 20% or an increase of 26%. For our specific benchmark, it’s approximately 88.14 min minus 69.75 min = 18.39 min, which means an improvement of 18 min. The 20% doesn’t look very exciting. Maybe you prefer to say: “It took 18 minutes less time to build”.

Do you build on your own workstation? You mean you don’t have a build server farm? An Icecream cluster might be the solution for you and your colleagues.