A few years back, I ran across this post by Hallvard Vassbotn. (It’s a shame he stopped blogging, because he always had some very interesting stuff about the technical details of how stuff in Delphi works.) At the bottom was a paragraph that really fascinated me:
On a more technical level it suffices to say that they use custom and extremely compact and fast data structures, tricks and hacks to be able to represent millions and millions of objects within the constraints of a 32-bit Windows system. Throw in the use of Physical Address Extensions, storing per-class information in “virtual” class vars to reduce object instance size, creation of classes and their VMTs dynamically at runtime (!!), pointer packing, multithreading, the list just goes on and on.
I figured if Hallvard’s friend can create classes and their VMTs dynamically at runtime, then anyone could, with the right level of technical knowledge about how a VMT works. My CodeRage presentation in 2009 was about theoretical research in that area, demonstrating that it could be done. But then I just sorta sat on it. It was a cool trick, but I didn’t have anything useful I could do with it, and I had more important things to work on, like my game engine.
But a few months ago I started ran into some serious issues with making the scripting for my game engine work. I’ve been using PascalScript, but its support for units is badly broken (and there’s no good way to fix it without basically rearchitecting the whole system) and its GOTO support is just broken enough to cause lots of trouble for me. (I need to support GOTOs properly. Don’t ask.) Image may be NSFW.
Clik here to view.
So I looked at DWS, which has been getting a lot of hype lately with all the recent development. But it also has bad unit support, no GOTOs at all, and no way to save a compiled script. I raised each of these issues with Eric and he gave reasons why he has no intention to fix each of them.
There’s one interesting thing DWS can do, though: you can define new classes and use them in the script engine. Being able to do this in my game engine would allow for a lot of customization opportunities. But a bit of testing demonstrated that, like the holographic Professor Moriarty on Star Trek: The Next Generation, these newly-defined objects are purely virtual constructs that can’t step out into the real world. If you try to bring a reference to one of these script objects into Delphi-land, you get a nil instead. This greatly reduces the opportunity for customization.
So I figured I may as well try my hand at my own version, drawing on my class-creation research. For the last few months, with a lot of hard work (and a fair bit of technical assistance from Barry Kelly on the arcane details of the RTTI system) I’ve been putting together a new Object Pascal-based script compiler. It’s not completely finished yet, but the features that will set it apart are up and running:
- Unit-based compilation. All your code doesn’t have to be in one unit. It also doesn’t expect that you’re building a single script to be executed directly. You can do that, but you can also run a valid compile cycle made up entirely of one or more units, with no program unit and no main routine, and use it as a library, calling individual routines from the script engine. Units can be compiled and saved individually, similar to Delphi’s DCU system, and linked with other units to form a script program.
- Dynamic, RTTI-based creation of new classes. You can define a new class in the script engine and the compiler will generate a new VMT and all the extended RTTI necessary to use this class from native Delphi code just like any other. For example, you can create a class that descends from a native type, override a virtual method, pass an instance out of the script engine to a variable reference, call the virtual method, and the script engine will be invoked to run it. (This particular trick require the TMethodImplementation class to set up, so Delphi XE or later will be required.)
The plan is to use the RTTI capabilities to achieve the most seamless integration possible between scripts and native code. In my next few posts I’ll be going over some details as to how I made this all work.