Fun with Yocto Project® “cross-prelink”

Objectives

This post shows you what’s the state with cross-prelink and the Yocto Project®. What is it supposed to do? Does it? What are the issues? What are the benefits?

Prerequisites

Cross Prelink

The latest version of the original prelink program is from 2013-05-03. Anyways this would only run on the host or the target system. It doesn’t work in a cross-build environment like the Yocto Project®. That’s why we use cross-prelink. The last change of the version used is from 2018-10-12.

Features

Cross-prelink promises run-time linking optimizations which lead to faster load times and less memory used.

• faster load times
• less memory used (less memory relocation – copy on write)
• less CPU used
• faster boot times
• less power consumption
• optional Address Space Layout Randomization (ASLR)

Let’s ignore ASLR from “prelink“, since we have more advanced methods to deal with ASLR. After running “prelink” or “cross-prelink” over an executable or a library it will slightly increase in size because a “.gnu.prelink_undo” is being added. Like this we are able to undo whatever “prelink” did. Keep that in mind, since I’m going to use this piece of information to determine whether “prelink” was able to modify a binary or not.

What’s wrong with cross-prelink?

Except for the fact that it’s code is not maintained? Initial tests show: “cross-prelink” does not do anything useful anymore. So it might be time to throw it out, just like many other distros did. A key phrase in the post of my friend Alex is this: “Recent tests show no difference with and without a “prelink“. And without a regression test it is very hard to say when and how this happened.[0][1]” I always like a good challenge. Especially if someone like Alex says it’s very hard. I will try to dig a bit deeper and see what’s going on.

There is no maintainer for “prelink”. This leaves the (management) question open who will fix a bug like this[3]. Let me mention here, that I also have an issue. It boils down to the fact that “cross-prelink” destroys third-party binaries (grafana). But my fix for this is just simple: Don’t use “cross-prelink” with these binaries.

PIE

Address Space Layout Randomization (ASLR) was introduced in the Linux kernel around 2001. ASLR makes it possible that with every reboot and program start the address spaces where the program is loaded into are not predictable anymore but randomized. This makes it harder to write exploits.

First, there was position independent code (PIC). Shared objects “.so” were built with PIC. This made ASLR possible, but only on libraries. Not on the executables.

Then, there was position-independent executable (PIE) mode, which allows not only libraries but also executables to be placed at randomized addresses in memory.

Keep in mind, that “PIE mode” was created to allow for ASLR.

PIE/ASLR and prelink

For more details check here. But remember: position-independent executables are not prelinkable. Or in other words: “cross-prelink” cannot be applied to binaries created in PIE mode.

The Yocto Project® and PIE mode

“PIE mode” was introduced around 2017 and you can find it in the release notes of Yocto Project® 2.4 (rocko 18.0.0) (search for “knob”). Since then it is the default on most architectures. Four years and eleven Releases later someone finds out: “prelink” does not work as expected. Even in theory it is not supposed to work with “PIE mode”. Apparently, people don’t really care too much about what “prelink” has to offer. Does it have anything to offer?

prelink: glibc 2.36 drops support for it

Support for “Prelink” is being dropped by glibc starting from version 2.36. It already causes issues with binary corruption, has a number of open bugs, and is of questionable benefit without disabling load address randomization and PIE executables. Prelink was disabled by default from Honister (October 2021) but people were still able to use it. Now we would be unable to maintain it without glibc support so the remaining pieces were removed. See this patch: [OE-core] [PATCH 1/2] prelink: Drop support for it

This is a local.conf with “prelink” enabled:

#
# Additional image features
#
# The following is a list of additional classes to use when building images which
# enable extra features. Some available options which can be included in this variable
# are:
#   - 'buildstats' collect build statistics
#   - 'image-mklibs' to reduce shared library files size for an image
#   - 'image-prelink' in order to prelink the filesystem image
# NOTE: if listing mklibs & prelink both, then make sure mklibs is before prelink
# NOTE: mklibs also needs to be explicitly enabled for a given image, see local.conf.extended
USER_CLASSES += "image-prelink"

This is a local.conf with “prelink” disabled:

#
# Additional image features
#
# The following is a list of additional classes to use when building images which
# enable extra features. Some available options which can be included in this variable
# are:
#   - 'buildstats' collect build statistics
# USER_CLASSES += "image-prelink"

After removing prelink-native we’ll also need the following patch:

author	Alexander Kanavin <alex.kanavin@gmail.com>	2022-01-27 11:20:05 +0100
committer	Richard Purdie <richard.purdie@linuxfoundation.org>	2022-02-01 07:31:18 +0000
commit	1adbf5ba2727f0023f31720e0ac7bcb2e67f039b (patch)
tree	2ddb88a41d8dd4123a68466456d7983054db6632
parent	8b562100df06f4655eb377ec259b58b10a13abee (diff)
download	poky-contrib-1adbf5ba2727f0023f31720e0ac7bcb2e67f039b.tar.gz
poky-contrib-1adbf5ba2727f0023f31720e0ac7bcb2e67f039b.tar.bz2
poky-contrib-1adbf5ba2727f0023f31720e0ac7bcb2e67f039b.zip
gobject-introspection: replace prelink-rtld with objdump -p
g-i internally processes the output with regexes, and seems
happy with what objdump is printing. It only needs to resolve
the library name as passed to the linker to the library file name.
Also recursive resolution (that ldd is doing and objdump is not)
is not necessary.

(From OE-Core rev: 767e0880d4d729e659e859dd99c1cdb084b8ba51)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat
-rw-r--r--	meta/classes/gobject-introspection.bbclass	2	
-rw-r--r--	meta/recipes-gnome/gobject-introspection/gobject-introspection/0001-giscanner-ignore-error-return-codes-from-ldd-wrapper.patch	28	
-rw-r--r--	meta/recipes-gnome/gobject-introspection/gobject-introspection_1.70.0.bb	14	
3 files changed, 3 insertions, 41 deletions
diff --git a/meta/classes/gobject-introspection.bbclass b/meta/classes/gobject-introspection.bbclass
index 4db1b362d9..7bf9feb0d6 100644
--- a/meta/classes/gobject-introspection.bbclass
+++ b/meta/classes/gobject-introspection.bbclass
@@ -29,7 +29,7 @@ EXTRA_OEMESON:prepend:class-nativesdk = "${@['', '${GIRMESONBUILD}'][d.getVar('G
 
 # Generating introspection data depends on a combination of native and target
 # introspection tools, and qemu to run the target tools.
-DEPENDS:append:class-target = " gobject-introspection gobject-introspection-native qemu-native prelink-native"
+DEPENDS:append:class-target = " gobject-introspection gobject-introspection-native qemu-native"
 
 # Even though introspection is disabled on -native, gobject-introspection package is still
 # needed for m4 macros.
diff --git a/meta/recipes-gnome/gobject-introspection/gobject-introspection/0001-giscanner-ignore-error-return-codes-from-ldd-wrapper.patch b/meta/recipes-gnome/gobject-introspection/gobject-introspection/0001-giscanner-ignore-error-return-codes-from-ldd-wrapper.patch
deleted file mode 100644
index b484b5e9e6..0000000000
--- a/meta/recipes-gnome/gobject-introspection/gobject-introspection/0001-giscanner-ignore-error-return-codes-from-ldd-wrapper.patch
+++ /dev/null
@@ -1,28 +0,0 @@
-From f742da8b3913f4818d3f419117076afe62f4dbf4 Mon Sep 17 00:00:00 2001
-From: Alexander Kanavin <alex.kanavin@gmail.com>
-Date: Wed, 5 Sep 2018 16:46:52 +0200
-Subject: [PATCH] giscanner: ignore error return codes from ldd-wrapper
-
-prelink-rtld, which we use instead of ldd returns 127 when it can't find a library.
-It is not an error per se, but it breaks subprocess.check_output().
-
-Upstream-Status: Inappropriate [oe-core specific]
-Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
-
----
- giscanner/shlibs.py | 2 +-
- 1 file changed, 1 insertion(+), 1 deletion(-)
-
-diff --git a/giscanner/shlibs.py b/giscanner/shlibs.py
-index d67df95..80352a6 100644
---- a/giscanner/shlibs.py
-+++ b/giscanner/shlibs.py
-@@ -102,7 +102,7 @@ def _resolve_non_libtool(options, binary, libraries):
-             args.extend(['otool', '-L', binary.args[0]])
-         else:
-             args.extend(['ldd', binary.args[0]])
--        output = subprocess.check_output(args)
-+        output = subprocess.run(args, check=False, stdout=subprocess.PIPE).stdout
-         if isinstance(output, bytes):
-             output = output.decode("utf-8", "replace")
- 
diff --git a/meta/recipes-gnome/gobject-introspection/gobject-introspection_1.70.0.bb b/meta/recipes-gnome/gobject-introspection/gobject-introspection_1.70.0.bb
index d4ee03d33c..4f72a33bfa 100644
--- a/meta/recipes-gnome/gobject-introspection/gobject-introspection_1.70.0.bb
+++ b/meta/recipes-gnome/gobject-introspection/gobject-introspection_1.70.0.bb
@@ -14,7 +14,6 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=c434e8128a68bedd59b80b2ac1eb1c4a \
                     "
 
 SRC_URI = "${GNOME_MIRROR}/${BPN}/${@oe.utils.trim_version("${PV}", 2)}/${BPN}-${PV}.tar.xz \
-           file://0001-giscanner-ignore-error-return-codes-from-ldd-wrapper.patch \
            file://0001-build-Avoid-the-doctemplates-hack.patch \
            "
 
@@ -33,9 +32,7 @@ DEPENDS += " libffi zlib glib-2.0 python3 flex-native bison-native autoconf-arch
 # target build needs qemu to run temporary introspection binaries created
 # on the fly by g-ir-scanner and a native version of itself to run
 # native versions of its own tools during build.
-# Also prelink-rtld is used to find out library dependencies of introspection binaries
-# (standard ldd doesn't work when cross-compiling).
-DEPENDS:append:class-target = " gobject-introspection-native qemu-native prelink-native"
+DEPENDS:append:class-target = " gobject-introspection-native qemu-native"
 
 # needed for writing out the qemu wrapper script
 export STAGING_DIR_HOST
@@ -55,13 +52,6 @@ EXTRA_OEMESON:class-target = " \
     ${@'-Dgir_dir_prefix=${libdir}' if d.getVar('MULTILIBS') else ''} \
 "
 
-# Need to ensure ld.so.conf exists so prelink-native works
-# both before we build and if we install from sstate
-do_configure[prefuncs] += "gobject_introspection_preconfigure"
-python gobject_introspection_preconfigure () {
-    oe.utils.write_ld_so_conf(d)
-}
-
 do_configure:prepend:class-native() {
         # Tweak the native python scripts so that they don't refer to the
         # full path of native python binary (the solution is taken from glib-2.0 recipe)
@@ -113,7 +103,7 @@ EOF
         # for a different architecture
         cat > ${B}/g-ir-scanner-lddwrapper << EOF
 #!/bin/sh
-prelink-rtld --root=$STAGING_DIR_HOST "\$@"
+$OBJDUMP -p "\$@"
 EOF
         chmod +x ${B}/g-ir-scanner-lddwrapper
 

Tests

Root file systems

We need to build four root file systems here:

# enable image-prelink
#USER_CLASSES ?= "buildstats image-prelink"
# disable image-prelink:
USER_CLASSES ?= "buildstats"

“PIE mode” is enabled by default. To disable it I did something like this:

# --> remove pie
GCCPIE = ""
GLIBCPIE = ""
SECURITY_CFLAGS_remove = "${SECURITY_PIE_CFLAGS}"
SECURITY_CFLAGS_pn-libgcc = ""
# <-- remove pie

To determine if files on the root file system are “pre-linked” or not I extract the rootfs tarball on the host and run this script:

#!/bin/sh

# That's what's in prelink.conf:

# -l /usr/local/sbin
# -l /sbin
# -l /usr/sbin
# -l /usr/local/bin
# -l /bin
# -l /usr/bin
# -l /usr/X11R6/bin
# -l /usr/games
# -l /usr/local/lib{,32,64,x32}
# -l /lib{,32,64,x32}
# -l /usr/lib{,32,64,x32}
# -l /usr/X11R6/lib{,32,64,x32}

# I assume there needs to be a prelink section to indicate that prelink was running

counter=0

if [ -f /tmp/elffiles ]; then
   rm -f /tmp/elffiles
fi

DIRS="usr/local/sbin sbin usr/sbin usr/local/bin bin usr/bin usr/X11R6/bin usr/games usr/local/lib lib usr/lib usr/X11R6/lib"

for directory in ${DIRS}; do
   if [ -d $directory ]; then
      # only do things if dir is not empty
      if [ "$(ls -A ${directory})" ]; then
         echo ">> ${directory}"
         find ${directory} -type f | xargs file | grep ELF | cut -f1 -d':' \
>> /tmp/elffiles
         echo "<< ${directory}"
      else
         echo "${directory} is empty"
      fi
   fi
done

for file in $(cat /tmp/elffiles) ; do
   #set -x
   readelf -S $file 2>/dev/null | grep -q prelink
   #set +x
   if [ $? -eq 0 ]; then
      echo "$file is prelinked"
      counter=$(($counter+1))
   fi
done

echo "${counter} files are prelinked"

prelinked-with-pie

$ ../../tests/check-prelink.sh 
>> sbin
<< sbin
>> usr/sbin
<< usr/sbin
>> bin
<< bin
>> usr/bin
<< usr/bin
usr/games is empty
>> lib
<< lib
>> usr/lib
<< usr/lib
usr/bin/grafana-cli is prelinked
usr/bin/grafana-server is prelinked
lib/libdl-2.33.so is prelinked
lib/ld-2.33.so is prelinked
lib/libpthread-2.33.so is prelinked
lib/libc-2.33.so is prelinked
6 files are prelinked

Please note, “usr/bin/grafana-cli” and “usr/bin/grafana-server” are third party binaries. There are destroyed by “cross-prelink“. Therefore they will not run but give a segmentation fault.

no-prelink-with-pie

$ ../../tests/check-prelink.sh 
>> sbin
<< sbin
>> usr/sbin
<< usr/sbin
>> bin
<< bin
>> usr/bin
<< usr/bin
usr/games is empty
>> lib
<< lib
>> usr/lib
<< usr/lib
0 files are prelinked

prelinked-no-pie

$ ../../tests/check-prelink.sh 
>> sbin
<< sbin
>> usr/sbin
<< usr/sbin
>> bin
<< bin
>> usr/bin
<< usr/bin
usr/games is empty
>> lib
<< lib
>> usr/lib
<< usr/lib
sbin/vipw.shadow is prelinked
sbin/fstab-decode is prelinked
sbin/mkfs.ext3 is prelinked
sbin/mkfs.ext4 is prelinked
sbin/halt.sysvinit is prelinked
sbin/mkfs.ext2.e2fsprogs is prelinked
sbin/sysctl.procps is prelinked
sbin/udevd is prelinked
sbin/nologin.shadow is prelinked
sbin/sulogin.util-linux is prelinked
sbin/init.sysvinit is prelinked
sbin/killall5 is prelinked
sbin/bootlogd is prelinked
sbin/shutdown.sysvinit is prelinked
sbin/runlevel.sysvinit is prelinked
sbin/mke2fs.e2fsprogs is prelinked
usr/sbin/chroot.coreutils is prelinked
usr/sbin/groupmod is prelinked
usr/sbin/groupadd is prelinked
usr/sbin/powerdebug is prelinked
usr/sbin/groupmems is prelinked
usr/sbin/start-stop-daemon.dpkg is prelinked
usr/sbin/grpconv is prelinked
usr/sbin/iw is prelinked
usr/sbin/avahi-daemon is prelinked
usr/sbin/lsof is prelinked
...
usr/lib/libxcb-dri2.so.0.0.0 is prelinked
usr/lib/libgio-2.0.so.0.6800.3 is prelinked
usr/lib/libxshmfence.so.1.0.0 is prelinked
usr/lib/libmicrohttpd.so.12.58.0 is prelinked
usr/lib/libprocps.so.8.0.3 is prelinked
usr/lib/libtirpc.so.3.0.0 is prelinked
usr/lib/libcairo.so.2.11600.0 is prelinked
usr/lib/libcrypt.so.2.0.0 is prelinked
usr/lib/libICE.so.6.3.0 is prelinked:
usr/lib/libperl.so.5.34.0 is prelinked
usr/lib/libxcb-sync.so.1.0.0 is prelinked
usr/lib/libunwind-arm.so.8.0.1 is prelinked
usr/lib/libxcb-present.so.0.0.0 is prelinked
usr/lib/libunwind.so.8.0.1 is prelinked
usr/lib/libXau.so.6.0.0 is prelinked
usr/lib/libpython3.9.so.1.0 is prelinked
usr/lib/libasm-0.185.so is prelinked
usr/lib/libxcb-xfixes.so.0.0.0 is prelinked
usr/lib/libreadline.so.8.1 is prelinked
usr/lib/libXcursor.so.1.0.2 is prelinked
usr/lib/libgdk-3.so.0.2404.25 is prelinked
usr/lib/libelf-0.185.so is prelinked
usr/lib/libX11.so.6.4.0 is prelinked
usr/lib/libXrandr.so.2.2.0 is prelinked
usr/lib/libglapi.so.0.0.0 is prelinked
usr/lib/libgmodule-2.0.so.0.6800.3 is prelinked
usr/lib/libatspi.so.0.0.1 is prelinked
usr/lib/libbfd-2.36.1.20210209.so is prelinked
usr/lib/libopcodes-2.36.1.20210209.so is prelinked
usr/lib/libdaemon.so.0.5.0 is prelinked
usr/lib/libXft.so.2.3.3 is prelinked
usr/lib/libxml2.so.2.9.12 is prelinked
usr/lib/libctf.so.0.0.0 is prelinked
usr/lib/libatk-bridge-2.0.so.0.0.0 is prelinked
usr/lib/libgtk-3.so.0.2404.25 is prelinked
usr/lib/libdw-0.185.so is prelinked
usr/lib/libXi.so.6.1.0 is prelinked
usr/lib/liblzma.so.5.2.5 is prelinked
usr/lib/libglib-2.0.so.0.6800.3 is prelinked
usr/lib/libgobject-2.0.so.0.6800.3 is prelinked
usr/lib/libsolv.so.1 is prelinked
usr/lib/libcairo-gobject.so.2.11600.0 is prelinked
usr/lib/libell.so.0.0.2 is prelinked
473 files are prelinked

no-prelink-no-pie

$ ../../tests/check-prelink.sh 
>> sbin
<< sbin
>> usr/sbin
<< usr/sbin
>> bin
<< bin
>> usr/bin
<< usr/bin
usr/games is empty
>> lib
<< lib
>> usr/lib
<< usr/lib
0 files are prelinked

Conclusion

In our specific case theory and practice seem to match. “Cross-prelink” does not show much of a difference since it can not operate on files compiled with PIE.

Appendix

If you want to learn how Embedded Linux works, click here. Here you can learn more about the Yocto Project®. For some more test and measurements with “cross-prelink” have a look here.