Using the Optimizing VC7 compiler with the VC6 IDE,with DEBUGGING

2009.09.18
By: morning.ye
you may contact him at: ywjheart ## gmail.com (replace ## to @)

from the source of atl/mfc in vc7x series,wo can see something interesting,that vc7 is able to output pdb files in the format used in vc6,the options is as following:
cl.exe:  /Zvc6
link.exe: /debugtype:vc6

however,vc7 toolset refuse the options if we add to commandline.
They lie to us,in fact,after debugging,we know M$ close the interface in commandline,but leave the functions in binaries.
OK,let's patch the toolset,in order to reenable the capability we need.

prepares:
1.install vc2003toolkit(any vc2003 edition is ok,and psdk/ddk 2003 also has these files),i'll use the toolset unpack from vc2003sp1,it's the lastest version.
copy the following files to C:\Program Files\Microsoft Visual Studio\VC98\bin\,don't forget to backup files from vc6!
msobj71.dll
c1.dll
c1xx.dll
c2.dll
cl.exe
link.exe
2. take a copy of the mspdb60.dll from PSDK/DDK2003,its version should equal to 6.2
3. hex editor

start patch:
1.1 modify the compiler front end
c1.dll in offset 78a11,change 0 to 1;
c1xx.dll in offset 18D3B1,change 0 to 1;
1.2 modify the compiler back end
c2.dll in offset 744C,change 1D to 3D;

2. modify the linker
link.exe
from 2a4dc to 2a4e7
original:  8A 98 60 01 00 00 84 DB
after patch: B3 01 88 98 60 01 00 00

3.open c1.dll,c1xx.dll,c2.dll,cl.exe,link.exe in a hex editor
replace all string "mspdb71.dll" to "mspdb60.dll"

4.in link.exe,search the string ?Open@PDB@@SAHPBD0KPAJQADPAPAU1@@Z ,and change it to ?Open@PDB@@SAHPAD0KPAJQADPAPAU1@@Z

5.copy the mspdb60.dll v6.2 to bin folder.

all done.now you can new a console project, and build the debug configuration,open the output file in a hex editor,you can find the  signature of  the old pdb format,"NB10",set a break in the function "main",and press F5, after break,you can see variables,callstack,and etc.

让VC7系列生成VC6支持PDB格式,以便VC6IDE可以调试
morning.ye 2009-9-18

我们知道 阅读VC7系列的 atl/mfc 源代码可以看到,VC7似乎具备生成 vc6 pdb文件的能力,其中参数分别是
cl.exe:  /Zvc6
link.exe: /debugtype:vc6

但是,很遗憾,如果我们把这个参数加上,它们纷纷表示,不认识.
当然,到底是否支持,调试一下才知道,通过分析,发现只是前端接口关掉了,不接受这些参数,但是内部实现确实支持的.
OK,patch它们.

准备工作:
拷贝一份vc2002/2003bin目录到vc6bin目录,把vc6的改名,vc2003toolkit也可以
从PSDK/DDK2003等地方拷贝 PDB60.dll,版本号应该是6.2

开始修改
1.修改编译前端
分别用od加载c1.dll,c2.dll,搜索字符串 pdb60.dll,在前方不远处有一个比较,记下这个全局变量的地址,在 LoadPE里转成 offset,并将原来的0改成1
c1.dll 在偏移10678a11处,将原来的0改成1
c1xx.dll 在偏移1058D3B1处,将原来的0改成1

2.修改连接器
link.exe
0042A4DC      8A98 60010000 mov     bl, byte ptr [eax+160]
0042A4E2      84DB          test    bl, bl
0042A4E4  |.  56            push    esi
0042A4E5  |.  57            push    edi
0042A4E6      74 0B         je      short link.0042A4F3
0042A4E8  |.  BF AC3C4000   mov     edi, link.00403CAC                   ;  mspdb60.dll
0042A4ED  |.  897C24 0C     mov     dword ptr [esp+C], edi
0042A4F1  |.  EB 0C         jmp     short link.0042A4FF
0042A4F3  |>  C74424 0C A03>mov     dword ptr [esp+C], link.00403CA0     ;  mspdb71.dll
0042A4FB  |.  8B7C24 0C     mov     edi, dword ptr [esp+C]
0042A4FF  |>  57            push    edi                                  ; /FileName
0042A500  |.  FF15 A4104000 call    dword ptr [<&KERNEL32.LoadLibraryA>] ; \LoadLibraryA

改成
0042A4DC      B3 01         mov     bl, 1              //把原来的判断改成赋值
0042A4DE      8898 60010000 mov     byte ptr [eax+160], bl      //
0042A4E4  |.  56            push    esi
0042A4E5  |.  57            push    edi
0042A4E6      90            nop                    //不再跳转,其实跳转也没关系,因为后面我们会替换这个名字
0042A4E7      90            nop
0042A4E8  |.  BF AC3C4000   mov     edi, link.00403CAC                         ;  mspdb60.dll
0042A4ED  |.  897C24 0C     mov     dword ptr [esp+C], edi
0042A4F1  |.  EB 0C         jmp     short link.0042A4FF
0042A4F3  |>  C74424 0C A03>mov     dword ptr [esp+C], link.00403CA0           ;  mspdb71.dll
0042A4FB  |.  8B7C24 0C     mov     edi, dword ptr [esp+C]
0042A4FF  |>  57            push    edi                                        ; /FileName
0042A500  |.  FF15 A4104000 call    dword ptr [<&KERNEL32.LoadLibraryA>]       ; \LoadLibraryA

3.用16进制编译器打开 c1.dll,c1xx.dll,c2.dll,cl.exe,link.exe
搜索 pdb71.dll 全部改成 pdb60.dll

4.在link.exe中把 ?Open@PDB@@SAHPBD0KPAJQADPAPAU1@@Z 改成 ?Open@PDB@@SAHPAD0KPAJQADPAPAU1@@Z
因为 pdb60.dll 把参数声明为 char*,而pdb71.dll把参数声明为 const char*,这导致修改符不同

5.从PSDK/DDK任何能找到 pdb60.dll的地方把 6.2 版的pdb60.dll 拷贝进去

OK,一切搞定,现在新建一个console工程,编译debug配置,可以看到 生成的文件内部已经是 NB10 标记的 PDB 文件了,开始调试,不再提示没有符号文件.

附注:
1.
如果提示找不到 ftol2

源代码中加上
extern "C" long _cdecl _ftol(double x);
long _cdecl _ftol2(double x){return _ftol(x);}

2.
关掉 /GZ,避免提示找不到 _SEH_epilog , _SEH_prolog
如果不关掉,可以从 PSDK2003 中拷贝sehprolg.obj,添加到工程

现在,可以继续使用 VC6这个轻快的IDE,配合VAX的书写代码提示,然后用VC7.1这个更标准的编译器编译了
还可以再拷贝一个 vcpkg,从而自动切换编译平台生成 i386/amd64 的目标文件

enjoy it!