Asan output not symbolized

Discussion:

Jeffrey Walton

2016-06-27 02:33:32 UTC

Hi Everyone,

I have a test script from help that repeatedly builds and runs a
library under different configurations. The script includes multiple
Asan tests.

The Asan tests are producing some findings under ARM32 as shown below.
Other platforms do not include Asan findings. In addition, Valgrind
does nt produce any findings.

The test program is always built with at least -g2, and sometimes
built with -g3. However, I am not seeing the symbolication. According
to the GCC folks, asan_symbolize is not required for GCC because it
uses libbacktrace. Also see
http://bugzilla.redhat.com/show_bug.cgi?id=1250844.

Why am I lacking symbolization, and how do I achieve it?

**********

AddressSanitizer: stack-buffer-overflow on address 0xbec57b18 at pc
0x38c651 bp 0xbec579e0 sp 0xbec579e4

AddressSanitizer: stack-buffer-overflow on address 0xbedbae9c at pc
0x6553f bp 0xbedbae68 sp 0xbedbae6c

AddressSanitizer: stack-buffer-overflow on address 0xbea67b18 at pc
0x38cbc5 bp 0xbea679e0 sp 0xbea679e4

AddressSanitizer: stack-buffer-overflow on address 0xbef0fe9c at pc
0x66117 bp 0xbef0fe68 sp 0xbef0fe6c

**********

$ uname -a
Linux cubietruck 3.4.39 #35 SMP PREEMPT Tue Sep 15 17:17:33 CST 2015
armv7l armv7l armv7l GNU/Linux

$ g++ --version
g++ (Ubuntu/Linaro 4.8.2-19ubuntu1) 4.8.2
Copyright (C) 2013 Free Software Foundation, Inc.

Maxim Ostapenko

2016-06-27 07:31:59 UTC

Permalink

Hi,

Post by Jeffrey Walton
Hi Everyone,
I have a test script from help that repeatedly builds and runs a
library under different configurations. The script includes multiple
Asan tests.
The Asan tests are producing some findings under ARM32 as shown below.
Other platforms do not include Asan findings. In addition, Valgrind
does nt produce any findings.
The test program is always built with at least -g2, and sometimes
built with -g3. However, I am not seeing the symbolication. According
to the GCC folks, asan_symbolize is not required for GCC because it
uses libbacktrace. Also see
http://bugzilla.redhat.com/show_bug.cgi?id=1250844.
Why am I lacking symbolization, and how do I achieve it?
**********
AddressSanitizer: stack-buffer-overflow on address 0xbec57b18 at pc
0x38c651 bp 0xbec579e0 sp 0xbec579e4
AddressSanitizer: stack-buffer-overflow on address 0xbedbae9c at pc
0x6553f bp 0xbedbae68 sp 0xbedbae6c
AddressSanitizer: stack-buffer-overflow on address 0xbea67b18 at pc
0x38cbc5 bp 0xbea679e0 sp 0xbea679e4
AddressSanitizer: stack-buffer-overflow on address 0xbef0fe9c at pc
0x66117 bp 0xbef0fe68 sp 0xbef0fe6c
**********

Are these full backtraces you got from ASan? If so, do you use stripped
libraries on your system (on Ubuntu, I guess, you do). Could this issue (
https://gcc.gnu.org/ml/gcc-patches/2016-03/msg00132.html) be related to to
your problem?

-Maxim

Post by Jeffrey Walton
$ uname -a
Linux cubietruck 3.4.39 #35 SMP PREEMPT Tue Sep 15 17:17:33 CST 2015
armv7l armv7l armv7l GNU/Linux
$ g++ --version
g++ (Ubuntu/Linaro 4.8.2-19ubuntu1) 4.8.2
Copyright (C) 2013 Free Software Foundation, Inc.
_______________________________________________
linaro-toolchain mailing list
https://lists.linaro.org/mailman/listinfo/linaro-toolchain

Jeffrey Walton

2016-07-01 22:55:32 UTC

Permalink

Post by Maxim Ostapenko

Post by Jeffrey Walton
The test program is always built with at least -g2, and sometimes
built with -g3. However, I am not seeing the symbolication. According
to the GCC folks, asan_symbolize is not required for GCC because it
uses libbacktrace. Also see
http://bugzilla.redhat.com/show_bug.cgi?id=1250844.
Why am I lacking symbolization, and how do I achieve it?

Are these full backtraces you got from ASan? If so, do you use stripped
libraries on your system (on Ubuntu, I guess, you do). Could this issue
(https://gcc.gnu.org/ml/gcc-patches/2016-03/msg00132.html) be related to to
your problem?

Not stripped; they are [supposed to be] fully symbolized. Here's the
CXXFLAGS used. Link occurs using $CXX with CXXFLAGS.

$ cat asan-output.txt
$ export CXXFLAGS="-DDEBUG -g3 -O0 -std=c++03 -mfpu=neon -fPIC
-fsanitize=address -fno-omit-frame-pointer -D_GLIBCXX_DEBUG"
...

We use one directory for *.h and *.cpp files. In addition, *.o files
go in the same directory. We do it to avoid problems with debuggers
finding symbols (it minimizes user questions).

Here's the full output, if interested: http://pastebin.com/Zn4ADDQ6.
It looks like there's some unbounded recursion going on, too.

Let me look into the bug, thanks.

Jeff

Jeffrey Walton

2016-07-01 23:06:32 UTC

Permalink

Post by Maxim Ostapenko

Post by Jeffrey Walton
AddressSanitizer: stack-buffer-overflow on address 0xbec57b18 at pc
0x38c651 bp 0xbec579e0 sp 0xbec579e4
AddressSanitizer: stack-buffer-overflow on address 0xbedbae9c at pc
0x6553f bp 0xbedbae68 sp 0xbedbae6c
AddressSanitizer: stack-buffer-overflow on address 0xbea67b18 at pc
0x38cbc5 bp 0xbea679e0 sp 0xbea679e4
AddressSanitizer: stack-buffer-overflow on address 0xbef0fe9c at pc
0x66117 bp 0xbef0fe68 sp 0xbef0fe6c

Are these full backtraces you got from ASan? If so, do you use stripped
libraries on your system (on Ubuntu, I guess, you do). Could this issue
(https://gcc.gnu.org/ml/gcc-patches/2016-03/msg00132.html) be related to to
your problem?

I don't believe the bug report applies...

Here's a crummy work around. First, install Clang on a machine, the do
a `which asan_symbolize` if you get lucky, it will be there (I found
one on my LeMaker HiKey). SCP it to the machine.

Then:

cubietruck:cryptopp$ ./cryptest.exe v 2>&1 | sed
's|/home/jwalton/cryptopp/||g' | asan_symbolize
Using seed: 1467414010

Testing Settings...

passed: Your machine is little endian.
passed: Your machine allows unaligned data access.
passed: sizeof(byte) == 1
passed: sizeof(word16) == 2
passed: sizeof(word32) == 4
passed: sizeof(word64) == 8
passed: sizeof(hword) == 2, sizeof(word) == 4, sizeof(dword) == 8
passed: hasNEON == 1, hasCRC32 == 0, hasAES == 0, hasSHA1 == 0, hasSHA2 == 0
=================================================================
==3719== ERROR: AddressSanitizer: stack-buffer-overflow on address
0xbebb9ac8 at pc 0x38b99d bp 0xbebb9990 sp 0xbebb9994
WRITE of size 4 at 0xbebb9ac8 thread T0
#0 0x38b99b in _ZN8CryptoPP6FilterC2EPNS_22BufferedTransformationE
/home/jwalton/cryptopp/filters.cpp:27
#1 0x38b99b in _ZN8CryptoPP6FilterC2EPNS_22BufferedTransformationE
/home/jwalton/cryptopp/filters.cpp:27
...

I could further pipe it through c++filt, but I have an idea of what to look for.

Jeff

Jeffrey Walton

2016-07-02 07:49:44 UTC

Permalink

Hi Maxim,

Post by Maxim Ostapenko
Are these full backtraces you got from ASan? If so, do you use stripped
libraries on your system (on Ubuntu, I guess, you do). Could this issue
(https://gcc.gnu.org/ml/gcc-patches/2016-03/msg00132.html) be related to to
your problem?

Maybe part of the intermediate solution is to provide asan_symbolize
since we have a partial work around. I suggest it because I know these
compiler problems can be a bear to track down.

If you want to provide use asan_symbolize for situations like these,
then I suggest either (1) running 2to3 on it; or (2) changing shebang
to '/usr/bin/env python2' .

If you run 2to3 on it, then test it. There will be two or three minor
issues to fix. Also see
http://github.com/google/sanitizers/issues/372.

I attached asan_symbolize from the LeMaker HiKey.

Jeff

Jeffrey Walton

2016-07-02 08:29:51 UTC

Permalink

Mystery solved... GCC 4.8 does not provide a symbolizer.

From a new comment on the GCC bug report

Post by Jeffrey Walton

I don't know what is different about things on my CubieTruck5 (Linaro, GCC
4.8.2), but I had to use asan_symbolize provided by Clang/Asan to get
symbols for a GCC/Asan finding.

Everything is different in this regard between 4.8.x and 4.9.x. Only the latter
has libbacktrace symbolization built in, the former doesn't have any
symbolization.

Adhemerval Zanella

2016-07-04 18:30:38 UTC

Permalink

Post by Jeffrey Walton

Mystery solved... GCC 4.8 does not provide a symbolizer.
From a new comment on the GCC bug report

Even with symbolizer you may face some issue with ASAN on ARM.
Check this llvm bug report [1].

[1] https://llvm.org/bugs/show_bug.cgi?id=22741

Post by Jeffrey Walton

I don't know what is different about things on my CubieTruck5 (Linaro, GCC
4.8.2), but I had to use asan_symbolize provided by Clang/Asan to get
symbols for a GCC/Asan finding.

Everything is different in this regard between 4.8.x and 4.9.x. Only the latter
has libbacktrace symbolization built in, the former doesn't have any
symbolization.

_______________________________________________
linaro-toolchain mailing list
https://lists.linaro.org/mailman/listinfo/linaro-toolchain

Jeffrey Walton

2016-07-04 18:56:55 UTC

Permalink

Post by Adhemerval Zanella

Post by Jeffrey Walton
Mystery solved... GCC 4.8 does not provide a symbolizer.
From a new comment on the GCC bug report

Even with symbolizer you may face some issue with ASAN on ARM.
Check this llvm bug report [1].
[1] https://llvm.org/bugs/show_bug.cgi?id=22741

Thanks, that makes sense. I could not understand why code that had
been stable for nearly 20 years was producing a finding under GCC 4.8
Asan.

The next mystery is why its producing a finding at all.

Jeff

Adhemerval Zanella

2016-07-04 19:35:37 UTC

Permalink

Post by Jeffrey Walton

Post by Adhemerval Zanella

Post by Jeffrey Walton
Mystery solved... GCC 4.8 does not provide a symbolizer.
From a new comment on the GCC bug report

Even with symbolizer you may face some issue with ASAN on ARM.
Check this llvm bug report [1].
[1] https://llvm.org/bugs/show_bug.cgi?id=22741

Thanks, that makes sense. I could not understand why code that had
been stable for nearly 20 years was producing a finding under GCC 4.8
Asan.
The next mystery is why its producing a finding at all.

The issue I referenced is about the stacktrace generated by
compiler-rt/libsanitizer when asan reports an error, not that asan is
generating a false-positive.