The llvmcompiler branch

Truecolor ZDoom with extra features and some unofficial/beta GZDoom features.
[Home] [Download] [Git builds (Win)] [Git builds (Mac)] [Libs (Win)] [Repo] [Bugs&Suggestions]

Moderators: Rachael, dpJudas

User avatar
Rachael
Developer
Developer
Posts: 3614
Joined: Sat May 13, 2006 10:30

Re: The llvmcompiler branch

Post by Rachael » Fri Sep 30, 2016 22:11

Honestly, I was very excited to see something new, and I did want to try this out. I am building on a bare-metal system but it is 8 years old and the entire thing also resides on a .vhd file (which should only affect performance by about 2-3% - but this is also an old 5900 RPM drive anyway).

I do not want to hold back progress. If there is any way to get this working, I want to go that route. Another thing I noticed (probably a bit too late now) is that it is building a Debug build instead of Release. >_> oops...

I did a check on how big it was, so far, and the entire tree is about 6-7 gigs in size.
Spoiler: Zen Sarcasm

dpJudas
Developer
Developer
Posts: 798
Joined: Sat Jul 23, 2016 7:53

Re: The llvmcompiler branch

Post by dpJudas » Fri Sep 30, 2016 22:35

Hehe, okay okay, I'll try see if I can make it bundle the LLVM headers and library files needed. The full compile of LLVM does include must-have things like PowerPC and AMDGPU opcodes. :D

User avatar
Graf Zahl
GZDoom Developer
GZDoom Developer
Posts: 7148
Joined: Wed Jul 20, 2005 9:48
Location: Germany
Contact:

Re: The llvmcompiler branch

Post by Graf Zahl » Fri Sep 30, 2016 22:47

The question should be, for which platforms precompiled binaries make sense. Windows, for sure, because it's the biggest piece. but what about others?

User avatar
Rachael
Developer
Developer
Posts: 3614
Joined: Sat May 13, 2006 10:30

Re: The llvmcompiler branch

Post by Rachael » Fri Sep 30, 2016 22:50

Mac. That goes without saying. But we're probably going to need help from _mental_ and Blzut3 for this.

Linux - as dpJudas said, it's likely in repos anyway so not a biggy.

ARM Linux - not too sure the route to go with that, it's likely there are precompiled libraries available for this platform, too, and it's popular enough to warrant consideration, but it is still so niche that it might not be worth the effort. (I do plan to get a Raspberry Pi in the coming weeks though - so I will be doing some investigating of my own on that anyway) - This has the added benefit of compatibility with Android GZDoom, though - as to how much interest in that there actually is, I don't know.

As far as Mac goes - I made a major investment about 8 years ago, buying a MacBook, with the sole intent of developing for that platform. I never got far with it. I plan to do that again, as the hardware by now is so obsolete it's close to useless anyway now.
Spoiler: Zen Sarcasm

dpJudas
Developer
Developer
Posts: 798
Joined: Sat Jul 23, 2016 7:53

Re: The llvmcompiler branch

Post by dpJudas » Fri Sep 30, 2016 22:58

There's a good chance that LLVM comes as part of Xcode on Mac - the primary compiler there is clang. I'll check this part a little later with my laptop.

User avatar
Rachael
Developer
Developer
Posts: 3614
Joined: Sat May 13, 2006 10:30

Re: The llvmcompiler branch

Post by Rachael » Sat Oct 01, 2016 0:32

There are many things that do not feel good - and having to delete an entire folder after 5 hours of building is one of them, but it was just too much space on the virtual drive and I couldn't afford it. It seems like I need a dedicated drive in order to even run this - it just bloats in size way too big. It really sucks that I had to do this.

Right now, I am just going to update QZDoom's source tree from G/ZDoom and post another build from the main branch. I will be gone most of the day, tomorrow, when I get more money in a week or so I will see about investing in a bigger drive.

Like I said, I don't want to hinder progress ... I don't want to give this up, yet. But there is no reasonable way we can expect this to work where someone has to download and compile the whole thing from scratch and after compile it would take up just as much room as a AAA game from 3 years ago would take.

There has to be a way we can have our cake and eat it, too - there has to be a way that we can still make use of this, without forcing someone else to go through this arduous process just to make their own builds.
Spoiler: Zen Sarcasm

dpJudas
Developer
Developer
Posts: 798
Joined: Sat Jul 23, 2016 7:53

Re: The llvmcompiler branch

Post by dpJudas » Sat Oct 01, 2016 1:03

I have a working version on my computer where it uses a folder with just the include files and a subset of the library files. Compressed with 7z it's 22 megabytes per build type, and 210 megabytes uncompressed. As we need four variations (debug, release, 32 bit and 64 bit) that's 800 MB for all of them with a 80 MB download.

Checking currently if there's any MS tool that can further strip such lib files.

User avatar
Rachael
Developer
Developer
Posts: 3614
Joined: Sat May 13, 2006 10:30

Re: The llvmcompiler branch

Post by Rachael » Sat Oct 01, 2016 1:35

You don't have to go that far. I would take an 80MB download->800MB uncompressed over 20 gb ANY day.
Spoiler: Zen Sarcasm

dpJudas
Developer
Developer
Posts: 798
Joined: Sat Jul 23, 2016 7:53

Re: The llvmcompiler branch

Post by dpJudas » Sat Oct 01, 2016 1:58

Yeah I guess 80 MB isn't a completely unreasonable download in 2016. Not when people watch movies and my Nvidia driver is something like 400 megs. How to distribute it? Upload a llvm.7z to the repository?

User avatar
Rachael
Developer
Developer
Posts: 3614
Joined: Sat May 13, 2006 10:30

Re: The llvmcompiler branch

Post by Rachael » Sat Oct 01, 2016 2:00

Upload it to a third-party host, for now. I'll put it on DRD Team, somewhere, and put a link to it.
Spoiler: Zen Sarcasm

User avatar
Graf Zahl
GZDoom Developer
GZDoom Developer
Posts: 7148
Joined: Wed Jul 20, 2005 9:48
Location: Germany
Contact:

Re: The llvmcompiler branch

Post by Graf Zahl » Sat Oct 01, 2016 6:49

I guess you could split it in two: a 32 bit and a 64 bit download. Most people who want to compile the source probably only need one architecture, saving them half of that large chunk.

dpJudas
Developer
Developer
Posts: 798
Joined: Sat Jul 23, 2016 7:53

Re: The llvmcompiler branch

Post by dpJudas » Sat Oct 01, 2016 8:38

Hmm yes, probably better to offer it as two individual downloads.

Since my last post I've discovered that if I build for Release rather than RelWithDebugInfo the size of the .lib files themselves dropped significantly. As we don't really need debug info for LLVM, even with debug builds, I've opted for compiling a release build with the debug C++ runtime instead for debug builds. That also has the advantage that ZDoom debug builds don't run slower when doing LLVM stuff.

That brings the file sizes down to the following:

llvm-3.9.0-everything.7z = 57 megabytes. This includes every library in LLVM and serves as backup just in case we suddenly need one more of the libs (recompiling all 4 buid types takes forever).
llvm-3.9.0-32bit.7z = 15 megabytes
llvm-3.9.0-64bit.7z = 24 megabytes

Uncompressed the 32bit and 64bit packages are in the 100-200 mb range each. I'll upload them somewhere as soon as I've confirmed I could build all four build types on my own computer first.

User avatar
Graf Zahl
GZDoom Developer
GZDoom Developer
Posts: 7148
Joined: Wed Jul 20, 2005 9:48
Location: Germany
Contact:

Re: The llvmcompiler branch

Post by Graf Zahl » Sat Oct 01, 2016 8:46

What does this contain? For example, for the scripting stuff I'd really like to have the option of plugging in some optimizers. These are in the 'everything' package, right?
Yes, and compiling this project takes forever. I just finished a 64 bit full debug build, it compiled for more than an hour.

But the actual problem here is that their project is just not presented well so that it can be easily tailored for such specific needs where someone just wants to do some JITing of script code but does not need all the bloat like code generators for foreign architectures, all the test stuff and what else.

dpJudas
Developer
Developer
Posts: 798
Joined: Sat Jul 23, 2016 7:53

Re: The llvmcompiler branch

Post by dpJudas » Sat Oct 01, 2016 9:05

They clearly originally intended to make it modular with a bunch of libraries where you could cherry-pick what you wanted. But somewhere down the line something went terribly wrong where half the libraries cross reference each other for literally ten lines of code. As a result, even the reduced package I created has almost all of the passes. It is mostly the exotic targets and such that are left out in the smaller 7z files.

The everything package has all the stuff yes. I made it just in case there should be some pass or feature I managed to not include in the reduced package that we want. Really don't want to compile this again just because I deleted one .lib file too many. :D

I think part of what may have gone wrong was that Apple more or less bought the entire LLVM development team to create clang. At that point they probably got too many 9-5 developers that had some deadline to meet and was more focused on clang performance than the LLVM modularity. Still, once you have LLVM linking, it opens up a lot of possibilities for teams that aren't big enough to write a fast compiler backend.

User avatar
Graf Zahl
GZDoom Developer
GZDoom Developer
Posts: 7148
Joined: Wed Jul 20, 2005 9:48
Location: Germany
Contact:

Re: The llvmcompiler branch

Post by Graf Zahl » Sat Oct 01, 2016 9:12

dpJudas wrote:I think part of what may have gone wrong was that Apple more or less bought the entire LLVM development team to create clang. At that point they probably got too many 9-5 developers that had some deadline to meet and was more focused on clang performance than the LLVM modularity. Still, once you have LLVM linking, it opens up a lot of possibilities for teams that aren't big enough to write a fast compiler backend.

Sounds reasonable. Corporate bloat at its best. At one of my former employers we had once developed an IDE for making mobile games once, it was also nicely modular in design but ended up one huge clusterfuck in the end because the people in charge had no idea how to properly maintain such a project. And the ultimate result was that the modularity made working with the code unnecessarily hard because crossing the boundaries between modules was not well implemented. That resulted in all the command line tools being developed for it pulling in the entire bunch of libraries which completely negated their purpose.

Locked

Return to “QZDoom”

Who is online

Users browsing this forum: No registered users and 1 guest