Cross capable prelink: load time + memory consumed

Objectives

This is the continuation to this article here. The other post describes of a “cross-capable-prelink” (“cross-prelink” or “prelink-cross”). It was created and brought to you by the Yocto Project®. What we will see here is: What’s the purpose of “prelink“? This article shows to measure various things to see if “cross-prelink” holds its promises. This hopefully helps to clarify a few things on when to use it and when to avoid using it.

Prerequisites

It might help to better understand the measurements performed here after reading the first part.

Summary of our last post

How many prelinked files are inside our four root file systems?

Load times

To measure load times first of all we need an executable that loads many shared objects (shared objects)”.so“. “/usr/bin/evim” was the executable I picked. It is a symlink to “/usr/bin/vim.vim” and it might be a good choice to start with. Below you can see how many “.so” files “vim.vim” needs:

ldd /usr/bin/evim
        linux-vdso.so.1 (0xbea8c000)
        libgtk-3.so.0 => /usr/lib/libgtk-3.so.0 (0xb6834000)
        libgdk-3.so.0 => /usr/lib/libgdk-3.so.0 (0xb6787000)
        libpangocairo-1.0.so.0 => /usr/lib/libpangocairo-1.0.so.0 (0xb676d000)
        libpango-1.0.so.0 => /usr/lib/libpango-1.0.so.0 (0xb6727000)
        libcairo.so.2 => /usr/lib/libcairo.so.2 (0xb6657000)
        libgdk_pixbuf-2.0.so.0 => /usr/lib/libgdk_pixbuf-2.0.so.0 (0xb6629000)
        libgobject-2.0.so.0 => /usr/lib/libgobject-2.0.so.0 (0xb65de000)
        libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0xb64ed000)
        libSM.so.6 => /usr/lib/libSM.so.6 (0xb64d7000)
        libICE.so.6 => /usr/lib/libICE.so.6 (0xb64b6000)
        libXt.so.6 => /usr/lib/libXt.so.6 (0xb6469000)
        libX11.so.6 => /usr/lib/libX11.so.6 (0xb6374000)
        libm.so.6 => /lib/libm.so.6 (0xb630e000)
        libtinfo.so.5 => /lib/libtinfo.so.5 (0xb62e3000)
        libacl.so.1 => /usr/lib/libacl.so.1 (0xb62cd000)
        libdl.so.2 => /lib/libdl.so.2 (0xb62ba000)
        libc.so.6 => /lib/libc.so.6 (0xb61bf000)
        /lib/ld-linux-armhf.so.3 (0xb6fbe000)
        libXrandr.so.2 => /usr/lib/libXrandr.so.2 (0xb61a8000)
        libXcursor.so.1 => /usr/lib/libXcursor.so.1 (0xb6191000)
        libXext.so.6 => /usr/lib/libXext.so.6 (0xb6176000)
        librt.so.1 => /lib/librt.so.1 (0xb6160000)
        libgmodule-2.0.so.0 => /usr/lib/libgmodule-2.0.so.0 (0xb614d000)
        libXi.so.6 => /usr/lib/libXi.so.6 (0xb6132000)
        libXcomposite.so.1 => /usr/lib/libXcomposite.so.1 (0xb611f000)
        libXdamage.so.1 => /usr/lib/libXdamage.so.1 (0xb610c000)
        libXfixes.so.3 => /usr/lib/libXfixes.so.3 (0xb60f8000)
        libcairo-gobject.so.2 => /usr/lib/libcairo-gobject.so.2 (0xb60e2000)
        libatk-1.0.so.0 => /usr/lib/libatk-1.0.so.0 (0xb60b9000)
        libatk-bridge-2.0.so.0 => /usr/lib/libatk-bridge-2.0.so.0 (0xb6086000)
        libxkbcommon.so.0 => /usr/lib/libxkbcommon.so.0 (0xb6041000)
        libwayland-cursor.so.0 => /usr/lib/libwayland-cursor.so.0 (0xb602a000)
        libwayland-egl.so.1 => /usr/lib/libwayland-egl.so.1 (0xb6018000)
        libwayland-client.so.0 => /usr/lib/libwayland-client.so.0 (0xb5fff000)
        libepoxy.so.0 => /usr/lib/libepoxy.so.0 (0xb5f3a000)
        libfribidi.so.0 => /usr/lib/libfribidi.so.0 (0xb5f11000)
        libgio-2.0.so.0 => /usr/lib/libgio-2.0.so.0 (0xb5ddc000)
        libpangoft2-1.0.so.0 => /usr/lib/libpangoft2-1.0.so.0 (0xb5dbd000)
        libharfbuzz.so.0 => /usr/lib/libharfbuzz.so.0 (0xb5d13000)
        libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0xb5cd4000)
        libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0xb5c5a000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xb5c35000)
        libpixman-1.so.0 => /usr/lib/libpixman-1.so.0 (0xb5bab000)
        libpng16.so.16 => /usr/lib/libpng16.so.16 (0xb5b78000)
        libxcb-shm.so.0 => /usr/lib/libxcb-shm.so.0 (0xb5b65000)
        libxcb.so.1 => /usr/lib/libxcb.so.1 (0xb5b3b000)
        libxcb-render.so.0 => /usr/lib/libxcb-render.so.0 (0xb5b21000)
        libXrender.so.1 => /usr/lib/libXrender.so.1 (0xb5b0a000)
        libz.so.1 => /lib/libz.so.1 (0xb5aea000)
        libGL.so.1 => /usr/lib/libGL.so.1 (0xb5a8d000)
        libjpeg.so.62 => /usr/lib/libjpeg.so.62 (0xb5a4a000)
        libffi.so.8 => /usr/lib/libffi.so.8 (0xb5a2f000)
        libpcre.so.1 => /usr/lib/libpcre.so.1 (0xb59cf000)
        libattr.so.1 => /usr/lib/libattr.so.1 (0xb59bb000)
        libdbus-1.so.3 => /usr/lib/libdbus-1.so.3 (0xb5978000)
        libatspi.so.0 => /usr/lib/libatspi.so.0 (0xb5942000)
        libmount.so.1 => /lib/libmount.so.1 (0xb58f4000)
        libresolv.so.2 => /lib/libresolv.so.2 (0xb58d4000)
        libexpat.so.1 => /usr/lib/libexpat.so.1 (0xb58a9000)
        libuuid.so.1 => /usr/lib/libuuid.so.1 (0xb5893000)
        libXau.so.6 => /usr/lib/libXau.so.6 (0xb5880000)
        libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0xb586b000)
        libglapi.so.0 => /usr/lib/libglapi.so.0 (0xb582c000)
        libdrm.so.2 => /usr/lib/libdrm.so.2 (0xb580d000)
        libxcb-glx.so.0 => /usr/lib/libxcb-glx.so.0 (0xb57eb000)
        libX11-xcb.so.1 => /usr/lib/libX11-xcb.so.1 (0xb57d9000)
        libxcb-dri2.so.0 => /usr/lib/libxcb-dri2.so.0 (0xb57c5000)
        libXxf86vm.so.1 => /usr/lib/libXxf86vm.so.1 (0xb57b1000)
        libxcb-dri3.so.0 => /usr/lib/libxcb-dri3.so.0 (0xb579d000)
        libxcb-present.so.0 => /usr/lib/libxcb-present.so.0 (0xb578a000)
        libxcb-sync.so.1 => /usr/lib/libxcb-sync.so.1 (0xb5775000)
        libxshmfence.so.1 => /usr/lib/libxshmfence.so.1 (0xb5763000)
        libxcb-xfixes.so.0 => /usr/lib/libxcb-xfixes.so.0 (0xb574d000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb5724000)
        libblkid.so.1 => /lib/libblkid.so.1 (0xb56dd000)

The test case which tries to measure load times looks like this:

#!/bin/sh
TEST="/usr/bin/evim"
set -x
md5sum ${TEST}
ldd ${TEST}
readelf -S ${TEST}
sync; echo 3 > /proc/sys/vm/drop_caches
perf stat sh -c "LD_BIND_NOW=true LD_DEBUG=statistics /usr/bin/evim 2>&1 > /dev/null"
set +x

If we put the results of our four root file systems in a table it looks like this:

How much free memory do we have after booting?

This is the test case I run to check for used memory:

#!/bin/sh
set -x
cat /proc/meminfo
procrank
cat /proc/slabinfo
set +x

The results from the test above when you run them on all four root file systems are this:

MemFree

“MemFree” (kB) was obtained by “cat /proc/meminfo” and describes free physical memory in both user and kernel space.

Interpretation of relative Numbers

We’ll look at whether there is more or less physical memory consumed. Less physical memory consumed is better. Please note that we typically access virtual memory in Linux and avoid direct physical memory access.

Third worst or third best, however you want to see it, is currently our default setting in the Yocto Project®. Please keep in mind, that most libraries are not “prelinked” but some crucial ones are (lib/libdl-2.33.so, lib/ld-2.33.so, lib/libpthread-2.33.so, lib/libc-2.33.so). Not “prelinked” somewhat implies that they were not compiled in “PIE mode” and hence they are a security issue.

Interpretation of absolute Numbers

Let’s say that more free physical memory is better, although we tyCross capable prelink:pically only access virtual memory.

MemAvailable

“MemAvailable” (kB) was obtained by “cat /proc/meminfo” and is an estimate of how much virtual memory is available after reclaiming it (caches, buffers, slab,…). “swap” is not used.

Interpretation of relative Numbers

We’ll look at an estimate of whether there is more or less virtual memory available after it’s being reclaimed. More virtual memory available is better.

Interpretation of absolute Numbers

Let’s assume more virtual memory available after reclaiming is better.

Memory Consumption of a specific executable after boot

This is the file we’ll look into:

~# ldd /usr/sbin/ofonod
        linux-vdso.so.1 (0xbedb4000)
        libudev.so.1 => /lib/libudev.so.1 (0xb6f33000)
        libell.so.0 => /usr/lib/libell.so.0 (0xb6ede000)
        libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0xb6ded000)
        libdbus-1.so.3 => /usr/lib/libdbus-1.so.3 (0xb6daa000)
        libdl.so.2 => /lib/libdl.so.2 (0xb6d97000)
        libc.so.6 => /lib/libc.so.6 (0xb6c9c000)
        /lib/ld-linux-armhf.so.3 (0xb6f5a000)
        libpcre.so.1 => /usr/lib/libpcre.so.1 (0xb6c3c000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xb6c17000)

The results from the test above when you run them on all four root file systems are this:

Conclusion

We can clearly see: “cross-prelink” can only work on binaries that were not compiled in “PIE mode”. You might turn off “PIE mode” and make your system more vulnerable to attacks. Or selectively compile certain libraries and executables with or without “PIE mode”. Only that way “cross-prelink” could do it’s job. I guess, at least network-facing apps should be compiled in “PIE mode” for security reasons.

Appendix

Keep in mind, that we still have the initial issue, that “cross-prelink” runs on certain third-party binaries and destroys them. How to selectively disable “cross-prelink” on those specific files? This might be the topic of another article.

If you want to learn how Embedded Linux works have a look here. To learn more about the Yocto Project® have a look here.