Issue building busybox using aarch64-linux-gnu 2014.11

Discussion:

Kevin Petit

2015-04-15 16:51:14 UTC

Hi,

I (and several other people) am (are) having trouble to build busybox with the latest aarch64-linux-gnu 2014.11 binaries we got from linaro.org.

It seems none of the built-in library search paths (gcc -print-search-dirs) allow to find crt[1i].o. They seem to be expected in a multiarch location but aren't (libc/usr/lib64). We're working around this with a symlink. Is this a bug? Have you seen similar issues?

Here's a failing command-line and the associated error message:

/path/to/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc -Wall -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wno-format-security -Wdeclaration-after-statement -Wold-style-definition -fno-builtin-strlen -finline-limit=0 -fomit-frame-pointer -ffunction-sections -fdata-sections -fno-guess-branch-probability -funsigned-char -static-libgcc -falign-functions=1 -falign-jumps=1 -falign-labels=1 -falign-loops=1 -fno-unwind-tables -fno-asynchronous-unwind-tables -Os -static -o busybox_unstripped -Wl,--start-group applets/built-in.o archival/lib.a archival/libarchive/lib.a console-tools/lib.a coreutils/lib.a coreutils/libcoreutils/lib.a debianutils/lib.a e2fsprogs/lib.a editors/lib.a findutils/lib.a init/lib.a libbb/lib.a libpwdgrp/lib.a loginutils/lib.a mailutils/lib.a miscutils/lib.a modutils/lib.a networking/lib.a networking/libiproute/lib.a networking/udhcp/lib.a printutils/lib.a procps/lib.a runit/lib.a selinux/lib.a shell/lib.a sysklogd/lib.a util-linux/lib.a util-linux/volume_id/lib.a archival/built-in.o archival/libarchive/built-in.o console-tools/built-in.o coreutils/built-in.o coreutils/libcoreutils/built-in.o debianutils/built-in.o e2fsprogs/built-in.o editors/built-in.o findutils/built-in.o init/built-in.o libbb/built-in.o libpwdgrp/built-in.o loginutils/built-in.o mailutils/built-in.o miscutils/built-in.o modutils/built-in.o networking/built-in.o networking/libiproute/built-in.o networking/udhcp/built-in.o printutils/built-in.o procps/built-in.o runit/built-in.o selinux/built-in.o shell/built-in.o sysklogd/built-in.o util-linux/built-in.o util-linux/volume_id/built-in.o -Wl,--end-group -Wl,--start-group -lm -Wl,--end-group
==========
/work/integration/envs/latest/tools/gcc/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu/bin/../lib/gcc/aarch64-linux-gnu/4.9.3/../../../../aarch64-linux-gnu/bin/ld: cannot find crt1.o: No such file or directory
/work/integration/envs/latest/tools/gcc/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu/bin/../lib/gcc/aarch64-linux-gnu/4.9.3/../../../../aarch64-linux-gnu/bin/ld: cannot find crti.o: No such file or directory
collect2: error: ld returned 1 exit status
make: *** [busybox_unstripped] Error 1

Thank you!

Regards,
Kévin

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782

Jim Wilson

2015-04-15 17:50:13 UTC

Permalink

It seems none of the built-in library search paths (gcc –print-search-dirs)
allow to find crt[1i].o. They seem to be expected in a multiarch location
but aren’t (libc/usr/lib64). We’re working around this with a symlink. Is
this a bug? Have you seen similar issues?

I downloaded a toolchain from this page.
https://www.linaro.org/downloads/

It works fine for a simple example. I don't see anything in the
option list that would cause a problem. Busybox could perhaps be
setting environment variables to change compiler behaviour, that can
cause problems, but it is rare that people set gcc environment
variables. That isn't obvious from the bug report. I haven't built
busybox in a long time, so I don't know if it is doing anything
unusual offhand.

You can debug this yourself a bit by adding a -v option to see the
linker option that gcc is running. This should contain explicit paths
to the crt1.o and crti.o files. You can check to see if the paths are
correct and the files exist. If there environment variables set, they
should be in the -v output.

You can also add -Wl,--verbose to get verbose info from the linker.
This will show you the linker built-in paths, and what files the
linker is trying to open. For my simple testcase I see

attempt to open
/home/wilson/Downloads/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu/bin/../libc/usr/lib/../lib64/crt1.o
succeeded
/home/wilson/Downloads/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu/bin/../libc/usr/lib/../lib64/crt1.o
attempt to open
/home/wilson/Downloads/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu/bin/../libc/usr/lib/../lib64/crti.o
succeeded
/home/wilson/Downloads/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu/bin/../libc/usr/lib/../lib64/crti.o

In the gcc --print-search-dirs output, at the end, I see

/home/wilson/Downloads/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu/bin/../libc/lib/:/home/wilson/Downloads/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu/bin/../libc/usr/lib/

These are the paths used to find the 64-bit libraries, which are just
../lib64 from the 32-bit libraries.

Jim

Kevin Petit

2015-04-16 09:46:43 UTC

Permalink

Hi Jim,

Adding the suggested options reveals something interesting:

attempt to open crt1.o failed
attempt to open crti.o failed
attempt to open /work/integration/envs/latest/tools/gcc/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu/bin/../lib/gcc/aarch64-linux-gnu/4.9.3/crtbeginT.o succeeded
/work/integration/envs/latest/tools/gcc/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu/bin/../lib/gcc/aarch64-linux-gnu/4.9.3/crtbeginT.o

Post by Jim Wilson
These are the paths used to find the 64-bit libraries, which are just
../lib64 from the 32-bit libraries.

And that's our issue! We've had the brilliant idea of storing toolchains in git repositories (not pointing the finger at anyone :-)) but git doesn't store empty folders. That means the empty libc/usr/lib folder used to construct the 64-bit libraries path using ../lib64 isn't there. It also explains why it was working for the person who pushed the change: he still had the directory coming from the extracted tarball.

Thank you very much for your help.

Regards,
Kévin

-----Original Message-----
From: Jim Wilson [mailto:***@linaro.org]
Sent: 15 April 2015 18:50
To: Kevin Petit
Cc: linaro-***@lists.linaro.org; Ian Spray; Mark Hambleton; James King
Subject: Re: Issue building busybox using aarch64-linux-gnu 2014.11

Post by Jim Wilson
It seems none of the built-in library search paths (gcc –print-search-dirs)
allow to find crt[1i].o. They seem to be expected in a multiarch location
but aren’t (libc/usr/lib64). We’re working around this with a symlink. Is
this a bug? Have you seen similar issues?

Rob Savoye

2015-04-15 21:55:39 UTC

Permalink

Post by Kevin Petit
I (and several other people) am (are) having trouble to build busybox with the latest aarch64-linux-gnu 2014.11 binaries we got from linaro.org.

Kevin Petit

2015-04-16 09:59:46 UTC

Permalink

Hi Rob,

Thanks for the info and pointers. It's good to know what is distributed is now what has been validated :-).

I'll give the new binaries a go and let you know if I find any issue.

Regards,
Kévin

-----Original Message-----
From: Rob Savoye [mailto:***@linaro.org]
Sent: 15 April 2015 22:56
To: Kevin Petit; linaro-***@lists.linaro.org
Cc: Ian Spray; Mark Hambleton; James King
Subject: Re: Issue building busybox using aarch64-linux-gnu 2014.11

Post by Kevin Petit
I (and several other people) am (are) having trouble to build busybox with the latest aarch64-linux-gnu 2014.11 binaries we got from linaro.org.

Binary releases built prior to 2014.11 used crosstool-ng to build the
binaries, and that process applied a patch (note that this was after
validation was done) that added multilib/multiarch support. We are no
longer using crosstool-ng, nor are we applying an untested patch. There
has been quite a bit of work lately on building binary tarballs, so I'd
try a newer set. Grab the latest (built a few days ago) from
http://abe.tcwglab.linaro.org/binaries/2015.02/. These are about to get
uploaded to releases.linaro.org, this is the staging area. If you find
any issues with these binaries, please file a bug for "ABE" at
https://bugs.linaro.org/, or email me back.

- rob -

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782

Joel (Xi Zhou) Zhou

2015-04-16 15:35:52 UTC

Permalink

Hi Rob,

In previous email, you mentioned that the crosstool-ng is no longer used.
Could you share some information how do you build the toolchain now?

Thanks,
Joel

-----Original Message-----
From: linaro-toolchain [mailto:linaro-toolchain-***@lists.linaro.org] On Behalf Of Kevin Petit
Sent: Thursday, April 16, 2015 6:00 AM
To: Rob Savoye; linaro-***@lists.linaro.org
Cc: Ian Spray; Mark Hambleton; James King
Subject: RE: Issue building busybox using aarch64-linux-gnu 2014.11

Hi Rob,

Thanks for the info and pointers. It's good to know what is distributed is now what has been validated :-).

I'll give the new binaries a go and let you know if I find any issue.

Regards,
Kévin

-----Original Message-----
From: Rob Savoye [mailto:***@linaro.org]
Sent: 15 April 2015 22:56
To: Kevin Petit; linaro-***@lists.linaro.org
Cc: Ian Spray; Mark Hambleton; James King
Subject: Re: Issue building busybox using aarch64-linux-gnu 2014.11

Post by Kevin Petit
I (and several other people) am (are) having trouble to build busybox with the latest aarch64-linux-gnu 2014.11 binaries we got from linaro.org.

Jim Wilson

2015-04-16 16:31:49 UTC

Permalink

On Thu, Apr 16, 2015 at 8:35 AM, Joel (Xi Zhou) Zhou

Post by Joel (Xi Zhou) Zhou
In previous email, you mentioned that the crosstool-ng is no longer used.
Could you share some information how do you build the toolchain now?

We wrote a tool called ABE that we use for building toolchains. You
can see some documentation here.
https://wiki.linaro.org/ABE
There is also some info in the README file and in comments at the top
of abe.sh inside the abe source tree.

Jim

Rob Savoye

2015-04-16 17:07:22 UTC

Permalink

In previous email, you mentioned that the crosstool-ng is no longer used. Could you share some information how do you build the toolchain now?

We stopped using crosstool-ng last Fall, and switched to our new ABE
tool. ABE is designed to plugin to our automated build & validation
system, which crosstool-ng was not. ABE also builds binary releases. You
can get ABE from: http://git.linaro.org/toolchain/abe.git. There's some
docs at https://wiki.linaro.org/ABE. I'm the primary author, so you can
send bugs to me. :-) I'd prefer they go into our public bugzilla,
http://bugs.linaro.org under the ABE category.

- rob -

Joel (Xi Zhou) Zhou

2015-04-16 18:09:06 UTC

Permalink

Thank you, Rob!
We have a "GLIBC_2.14" not found issue on RHEL6 when using gcc-linaro-4.9-2014.11-x86_64_arm-linux-gnueabihf. I may need to re-build toolchain on RHEL6. When I hit the rock, I know who I can refer to;)

Joel
-----Original Message-----
From: Rob Savoye [mailto:***@linaro.org]
Sent: Thursday, April 16, 2015 1:07 PM
To: Joel (Xi Zhou) Zhou; Kevin Petit; linaro-***@lists.linaro.org
Cc: Ian Spray; Mark Hambleton; James King
Subject: Re: Issue building busybox using aarch64-linux-gnu 2014.11

In previous email, you mentioned that the crosstool-ng is no longer used. Could you share some information how do you build the toolchain now?

Rob Savoye

2015-04-16 18:15:21 UTC

Permalink

Post by Joel (Xi Zhou) Zhou
We have a "GLIBC_2.14" not found issue on RHEL6 when using gcc-linaro-4.9-2014.11-x86_64_arm-linux-gnueabihf. I may need to re-build toolchain on RHEL6. When I hit the rock, I know who I can refer to;)

The new binaries are built using LSB, so shouldn't have this problem
now. Let me know! You can also try using ABE to rebuild: "[blah]/abe.sh
--build all --target arch64-linux-gnu". If you add "--tarbin" to the
command line, it'll also build a fresh binary snapshot.

- rob -

luther.ge

2015-04-16 05:28:16 UTC

Permalink

Hi Kevin
it works fine in my side, the busybox & aarch64-linux-gnu version like
these:
1.
http://releases.linaro.org/14.11/components/toolchain/binaries/aarch64-linux-gnu/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu.tar.xz
2. git clone git://git.busybox.net/busybox
3. git log -n 1
***@gliethttp:/vobs/works/busybox$ git log -n 1
commit f7466e477691fd29f47ebe8ae27489e065c69e5e
Author: Felix Fietkau <***@openwrt.org>
Date: Thu Apr 9 10:20:16 2015 +0200

find: fix regression in status processing for path arguments

Regression added in commit 14158b4127dba30466c50147b868a6a89702960b
"find: add optional support for '-exec ... {} +'"

This commit causes find to exit on the first path argument that was not
found, which breaks existing scripts and is incompatible to other
implementations.

Instead of exiting on the first failure, return EXIT_FAILURE at the end
if any error occurred.

Signed-off-by: Felix Fietkau <***@openwrt.org>
Signed-off-by: Denys Vlasenko <***@googlemail.com>

4. CROSS_COMPILE=aarch64-linux-gnu- make ARCH=arm64 menuconfig
Busybox Settings
---> Build Options
---> [*] Build BusyBox as a static binary (no shared libs)
Busybox Settings
---> General Configuration
---> [*] Don't use /usr
5. CROSS_COMPILE=aarch64-linux-gnu- make ARCH=arm64 install

Post by Kevin Petit
Hi,
I (and several other people) am (are) having trouble to build busybox
with the latest aarch64-linux-gnu 2014.11 binaries we got from linaro.org.
It seems none of the built-in library search paths (gcc
–print-search-dirs) allow to find crt[1i].o. They seem to be expected in
a multiarch location but aren’t (libc/usr/lib64). We’re working around
this with a symlink. Is this a bug? Have you seen similar issues?
/path/to/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc
-Wall -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused
-Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes
-Wmissing-declarations -Wno-format-security
-Wdeclaration-after-statement -Wold-style-definition -fno-builtin-strlen
-finline-limit=0 -fomit-frame-pointer -ffunction-sections
-fdata-sections -fno-guess-branch-probability -funsigned-char
-static-libgcc -falign-functions=1 -falign-jumps=1 -falign-labels=1
-falign-loops=1 -fno-unwind-tables -fno-asynchronous-unwind-tables -Os
-static -o busybox_unstripped -Wl,--start-group applets/built-in.o
archival/lib.a archival/libarchive/lib.a console-tools/lib.a
coreutils/lib.a coreutils/libcoreutils/lib.a debianutils/lib.a
e2fsprogs/lib.a editors/lib.a findutils/lib.a init/lib.a libbb/lib.a
libpwdgrp/lib.a loginutils/lib.a mailutils/lib.a miscutils/lib.a
modutils/lib.a networking/lib.a networking/libiproute/lib.a
networking/udhcp/lib.a printutils/lib.a procps/lib.a runit/lib.a
selinux/lib.a shell/lib.a sysklogd/lib.a util-linux/lib.a
util-linux/volume_id/lib.a archival/built-in.o
archival/libarchive/built-in.o console-tools/built-in.o
coreutils/built-in.o coreutils/libcoreutils/built-in.o
debianutils/built-in.o e2fsprogs/built-in.o editors/built-in.o
findutils/built-in.o init/built-in.o libbb/built-in.o
libpwdgrp/built-in.o loginutils/built-in.o mailutils/built-in.o
miscutils/built-in.o modutils/built-in.o networking/built-in.o
networking/libiproute/built-in.o networking/udhcp/built-in.o
printutils/built-in.o procps/built-in.o runit/built-in.o
selinux/built-in.o shell/built-in.o sysklogd/built-in.o
util-linux/built-in.o util-linux/volume_id/built-in.o -Wl,--end-group
-Wl,--start-group -lm -Wl,--end-group
==========
cannot find crt1.o: No such file or directory
cannot find crti.o: No such file or directory
collect2: error: ld returned 1 exit status
make: *** [busybox_unstripped] Error 1
Thank you!
Regards,
Kévin
-- IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended
recipient, please notify the sender immediately and do not disclose the
contents to any other person, use it for any purpose, or store or copy
the information in any medium. Thank you.
ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ,
Registered in England & Wales, Company No: 2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1
9NJ, Registered in England & Wales, Company No: 2548782
_______________________________________________
linaro-toolchain mailing list
https://lists.linaro.org/mailman/listinfo/linaro-toolchain

Kevin Petit

2015-04-16 09:15:29 UTC

Permalink

Hi Luther,

Doing that separately works for me too on the same machine, even with our version of busybox (1_22_stable).

There must be something else somewhere. I'll keep digging.

Thanks for your help.

Regards,
Kévin

-----Original Message-----
From: luther.ge [mailto:***@linaro.org]
Sent: 16 April 2015 06:28
To: Kevin Petit; linaro-***@lists.linaro.org
Cc: Ian Spray; Mark Hambleton; James King
Subject: Re: Issue building busybox using aarch64-linux-gnu 2014.11

Hi Kevin
it works fine in my side, the busybox & aarch64-linux-gnu version like
these:
1.
http://releases.linaro.org/14.11/components/toolchain/binaries/aarch64-linux-gnu/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu.tar.xz
2. git clone git://git.busybox.net/busybox
3. git log -n 1
***@gliethttp:/vobs/works/busybox$ git log -n 1
commit f7466e477691fd29f47ebe8ae27489e065c69e5e
Author: Felix Fietkau <***@openwrt.org>
Date: Thu Apr 9 10:20:16 2015 +0200

find: fix regression in status processing for path arguments

Regression added in commit 14158b4127dba30466c50147b868a6a89702960b
"find: add optional support for '-exec ... {} +'"

This commit causes find to exit on the first path argument that was not
found, which breaks existing scripts and is incompatible to other
implementations.

Instead of exiting on the first failure, return EXIT_FAILURE at the end
if any error occurred.

Signed-off-by: Felix Fietkau <***@openwrt.org>
Signed-off-by: Denys Vlasenko <***@googlemail.com>

4. CROSS_COMPILE=aarch64-linux-gnu- make ARCH=arm64 menuconfig
Busybox Settings
---> Build Options
---> [*] Build BusyBox as a static binary (no shared libs)
Busybox Settings
---> General Configuration
---> [*] Don't use /usr
5. CROSS_COMPILE=aarch64-linux-gnu- make ARCH=arm64 install

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782

luther.ge

2015-04-16 09:19:29 UTC

Permalink

Kevin,
You are Welcome :-)

Post by Kevin Petit
Hi Luther,
Doing that separately works for me too on the same machine, even with our version of busybox (1_22_stable).
There must be something else somewhere. I'll keep digging.
Thanks for your help.
Regards,
Kévin
-----Original Message-----
Sent: 16 April 2015 06:28
Cc: Ian Spray; Mark Hambleton; James King
Subject: Re: Issue building busybox using aarch64-linux-gnu 2014.11
Hi Kevin
it works fine in my side, the busybox & aarch64-linux-gnu version like
1.
http://releases.linaro.org/14.11/components/toolchain/binaries/aarch64-linux-gnu/gcc-linaro-4.9-2014.11-x86_64_aarch64-linux-gnu.tar.xz
2. git clone git://git.busybox.net/busybox
3. git log -n 1
commit f7466e477691fd29f47ebe8ae27489e065c69e5e
Date: Thu Apr 9 10:20:16 2015 +0200
find: fix regression in status processing for path arguments
Regression added in commit 14158b4127dba30466c50147b868a6a89702960b
"find: add optional support for '-exec ... {} +'"
This commit causes find to exit on the first path argument that was not
found, which breaks existing scripts and is incompatible to other
implementations.
Instead of exiting on the first failure, return EXIT_FAILURE at the end
if any error occurred.
4. CROSS_COMPILE=aarch64-linux-gnu- make ARCH=arm64 menuconfig
Busybox Settings
---> Build Options
---> [*] Build BusyBox as a static binary (no shared libs)
Busybox Settings
---> General Configuration
---> [*] Don't use /usr
5. CROSS_COMPILE=aarch64-linux-gnu- make ARCH=arm64 install

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782