[split] Compiler off-loading discussion

[Rachael]

Curious but wouldn't it be better, perhaps, instead of providing .asm files, to compile the code directly into the executable? That way the source stays consistently C++ and LLVM is just used more or less as a bridge for the drawers.

I really don't mind having the LLVM dependency, but all things considered (and especially after playing with the LLVM target types) I'm thinking it would be better to have a single block of code that gets put together by the compiler and associated utilities rather than having to compile in the .asm files manually. That would also be beneficial for cross-architecture compiles - for example, compiling QZDoom for ARM devices. (ARMs have souls too, ya know! Even if you mostly hold it with your arm! .... okay I'll show myself out... sorry...)

Sorry, I am not trying to be nitpicky. I just dislike the idea of dealing with .asm files at all, and having the drawer code right at the gate seems to be more ideal for anyone trying to mod the engine, rather than someone who would have to eventually get LLVM anyway just to even make changes to the drawers.

[dpJudas]

Well, the .asm files gets linked into the executable - all of them. For example, there would be a DrawColumnSSE2, DrawColumnSSE4, DrawColumnAVX and DrawColumnAVX2. When QZDoom boots, it looks at the CPU and decides it can do SSE4. So it points DrawColumn to DrawColumnSSE4.

The drawergen tool could theoretically output .cpp files instead of .asm, but the catch here is that such .cpp files would have to be compiled with specific flags to inform the compiler that it can safely generate SSE and AVX opcodes. The actual contents of those .cpp files would probably not be much more readable than the corresponding .asm files. It would remove the need for LLVM, but using LLVM still has the advantage of being able to "talk" more directly to a compiler backend. Stuff like me already telling LLVM that there cannot possibly be any pointer aliasing going on between the texture pointers and the framebuffer pointers, or which for loops to unroll and by how much.

Writing the drawers as purely C++ code is not really an option. The LLVM drawers are somewhere between 50% to 100% faster than the SSE intrinsics versions that the truecolor branch had. Those, in turn, were twice as fast as pure C++ code without intrinsics. We also have something like 50 drawer variations at this point, which is one of the reasons the launch time has gone up as much as it has.

[Rachael]

What I mean is leaving the LLVM drawer code as-is, but instead of compiling at run-time, to compile it directly into the executable at compile time, and not have to embed the LLVM libraries themselves directly into the executable.

LLVM would be executed by MSVC++, and would simply compile the drawers into the 4 sets you mentioned and then MSVC++ would pick up the 4 generated .asm files, run NASM/YASM to get .obj files, and link them accordingly.

It's almost the same as your original proposal, except a: the .asm files would not exist in the actual source, and b: it would still improve boot times by offloading the compile to the compiler, rather than the user.

[dpJudas]

That's exactly what that the drawergen tool would do. It is the drawergen tool that needs LLVM, and not QZDoom. Either then run the drawergen tool as part of CMake, or just add the .asm files to the repository so only LLVM is needed if you need to update the drawers.

[Rachael]

Running drawergen as part of CMake sounds fine to me.

(I am *really* sorry if I seem so nit-picky >_<)

My goal with this suggestion is - if the person preparing the executable modifies any part of the original source - including the drawers - those changes are effected immediately without the need to manually run an intermediary tool (which would be run automatically by MSVC++, anyway).

[dpJudas]

Yeah, I guess it could build the drawer tool first, and then use what it built to generate .asm files (or .obj files directly, as theoretically LLVM can do that).

[Rachael]

I'm sorry. I am not trying to make this difficult for you. It was just a suggestion.

[dpJudas]

Actually, just discovered I could loot the updaterevision CMake files for this, so already half way done with that part of it.

What I'm really trying to figure out now is if it should output .asm or .obj files.

[Rachael]

Wouldn't .obj files be less work for you?

[dpJudas]

Trouble with .obj is that I don't know how to get LLVM to output that (although I'm 100% sure it can). It is probably the best option, but I don't like going through hell first. LLVM's documentation is virtually non-existent for its C++ API.

[Rachael]

Well - the .asm -> NASM path is fine even if not preferable - whatever works comes first over efficiency and cleanliness.

ZDoom may be eliminating its NASM dependency soon though, which will put QZDoom in a bind because that's one commit that it will have to delay.

[Graf Zahl]

Concerning assemblers, the only reason ZDoom uses NASM is because it's platform independent. If you can create files for both MASM and GAS it should work everywhere, as MASM is part of the VS distribution.

[dpJudas]

Turns out getting it to create the .obj file was a lot easier than I expected. The drawers are now generated at compile time rather than run time. It only creates the SSE 2 version for now, because from what I could tell it didn't really make any speed difference between that and the AVX-2 version. Good news is this doesn't require any assembler at all and the start time is now as fast as ZDoom.

The bad news is that the tool runs on every build, which is not really needed. I need to find some good way for it to know that the .obj file is already up to date and doesn't need a recompile.

[Rachael]

Is there anything special I need to do to get this working? Currently I am getting this linker error:

Code: Select all

3>LINK : fatal error LNK1181: cannot open input file 'P:\Users\Guest\Desktop\build\QZDoom\src\r_drawersasm.obj'

[dpJudas]

Hmm, it should be writing something like this earlier in the log:

Code: Select all

1>------ Build started: Project: drawergen_target, Configuration: RelWithDebInfo x64 ------
1>  Compiling drawer code for pentium4..

That step outputs that .obj file that it says it cannot find.