GSOC 2017 - Crash Reporting - Report 3

Posted 7 years ago

I decided to make a flow chart to show the various processes of breakpads and where they took place:

muse_breakpads.png

Next I made a Qt example based on the Qt breakpad integration example from here:
https://github.com/JPNaude/dev_notes/wiki/Using-Google-Breakpad-with-Qt

My source code of the example is locate here:
https://github.com/gsocnikos/musescore_crash/tree/master/qt_example/qt_…

My build is located here:
https://github.com/gsocnikos/musescore_crash/tree/master/qt_example/bui…

I compiled it using CMake and MSVC2015

When you run the exe it will show up a Qt window for a few seconds and after that the crash will take place where it creates a minidump file (.dmp) under your home directory, for me:
c:\Users\nickhatz\a32acd69-0d4a-4630-a1e8-6ebc6b7ac502.dmp

We need to generate the symbol file, from the git repository of breakpads the symdump it comes precompiled so I run this exe:
C:\Users\nickhatz\breakpad\src\tools\windows\binaries\dump_syms.exe qt_example.pdb > qt_example.sym
C:\Users\nickhatz\breakpad\src\tools\windows\binaries\dump_syms.exe C:\Qt\5.8\msvc2015\bin\Qt5Cored.pdb > Qt5Cored.sym
C:\Users\nickhatz\breakpad\src\tools\windows\binaries\dump_syms.exe C:\Qt\5.8\msvc2015\bin\Qt5Qmld.pdb > Qt5Qmld.sym

Then I moved dmp and sym files under my home directory under cygwin where I have compiled minidump_stackwalk (C:\cygwin\home\nickhatz\qt_example) and a created the symbols folder using ./create_sym_dir.sh qt_example.sym
./create_sym_dir.sh Qt5Cored.sym
./create_sym_dir.sh Qt5Qmld.sym

where it store the sym file as:
symbols/qt_example.pdb/E7638109039C47A28FAA206CE43475F5a/qt_example.sym
symbols/Qt5Cored.pdb/D88DE4555E7D46AA9267101BD67155E61/Qt5Cored.sym
symbols/Qt5Qmld.pdb/E15BC75299D4412AAB19044FCB3C02171/Qt5Qmld.sym

then I run the minidump_stackwalk using:
/usr/local/breakpads/bin/minidump_stackwalk.exe a32acd69-0d4a-4630-a1e8-6ebc6b7ac502.dmp symbols 1> stackwalk.txt 2> stackwalk.log

that creates the crash report at the file: stackwalk.txt
and some log indormation: stackwalk.log

from the stackwalk.txt we can identify the lines in our source of where the crash was located:

2 qt_example.exe!buggyFunc() [main.cpp : 9 + 0x1d]

3 qt_example.exe!main [main.cpp : 26 + 0x5]

The files from my breakpad reports are located here:
https://github.com/gsocnikos/musescore_crash/tree/master/qt_example/bre…

To compile minidump_stackwalk I used cygwin and I download breakpad code as:
git clone https://chromium.googlesource.com/breakpad
cd breakpad
./configure --prefix=/usr/local/breakpads
Then change from Makefile the flag: c++11 to gnu++11 (where it corrects an ANSI problem)
make && make install

If you are not interested in compiling it you can download it from here:
https://hg.mozilla.org/build/tools/file/tip/breakpad/win32


Comments

OK. Looks like you have a good understanding now. You also have a running example to create a pdb, generate a minidump (based on VC++- and decode it with minidump_stackwalk.

From there, I believe it's not a good use of your time to focus on the network interaction (upload of minidump, upload of symbols, presentation of the webpage for crash reports) but instead we need to make sure we gather useful data and that we can run the whole thing within MuseScore and not a simple example. So for the coming week, I would propose the following.

  1. Find out if the stacktrace are readable and helpful, in particular, can we get stacktrace down to libraries? Do we need pdb from Qt, other librairies? If yes, how do we get them?
  2. Find out if we can add metadata in the minidump or as a sidecar file. How do other project deal with that? When a crash happens in MuseScore, the minidump and stacktrace are interesting, but if we could attach the filename of the current score, the loaded soundfont, the time since MuseScore starts, the RAM usage etc... it would definitely help to debug the crash. See for example https://www.chromium.org/developers/crash-reports and https://www.chromium.org/developers/debugging-with-crash-keys
  3. It seems breakpad on windows works only if the client, in this case MuseScore, is compiled with VC++. That's a huge task but definitely worth doing. It will enable a lot more than just crash reporting.

I'd like to set up an environment in VS2019, but I've got an outside reference lnk2019.
Can I get the breakpad set by VS2019 by any chance?