Of course the Apple Network Server can be hacked into running Doom

Now, let's go ahead and get the grumbling out of the way. No, the ANS is not running Linux or NetBSD. No, this is not a backport of NCommander's AIX Doom, because that runs on AIX 4.3. The Apple Network Server could run no version of AIX later than 4.1.5 and there are substantial technical differences. (As it happens, the very fact it won't run on an ANS was what prompted me to embark on this port in the first place.) And no, this is not merely an exercise in flogging a geriatric compiler into building Doom Generic, though we'll necessarily do that as part of the conversion. There's no AIX sound driver for ANS audio, so this port is mute, but at the end we'll have a Doom executable that runs well on the ANS console under CDE and has no other system prerequisites. We'll even test it on one of IBM's PowerPC AIX laptops as well. Because we should. AIX ("Advanced Interactive Executive") ended up as the 1996 Apple Network Server's primary operating system almost by default, Apple's first true Unix server since the A/UX-based Workgroup Server 95, but IBM AIX has a long history of its own dating back to the 1986 IBM RT PC. That machine was based on the IBM ROMP CPU as derived from the IBM 801, generally considered the first modern RISC design. AIX started as an early port of UNIX System V Release 3 and is thus a true Unix descendent, but also merged in some code from BSD 4.2 and 4.3. The RT PC ran AIX v1 and v2 as its primary operating systems, though IBM also supported 4.3BSD (ported by IBM as the Academic Operating System) and a spin of Pick OS. Although a truly full-fledged workstation with aggressive support from IBM, it ended up a failure in the market due to comparatively poor performance and notorious problems with its floating point support.

Nevertheless, AIX's workstation roots persisted even through the substantial rewrite that became version 3 in 1989, and was likewise the primary operating system for its next-generation technical workstations now based on POWER. AIX 3 introduced AIXwindows, a licensed port of X.desktop from IXI Limited (later acquired by another licensee, SCO) initially based on X11R3 with Motif as the toolkit. In 1993 the "SUUSHI" partnership — named for its principals, Sun, Unix System Laboratories, the Univel joint initiative of Novell and AT&T, SCO, HP and IBM — negotiated an armistice in the Unix Wars, their previous hostilities now being seen as counterproductive against common enemy Microsoft. This partnership led to the Common Open Software Environment (COSE) initiative and the Common Desktop Environment (CDE), derived from HP VUE, which was also Motif-based. Coincidentially (probably), the first AIX release to feature CDE is 4.1.4, the first official release of AIX (as 4.1.4.1) to run on the ANS. (4.1.2 was only used on pre-release ANSes and never supported nor sold by Apple.) Having professionally worked with RS/6000 and Power ISA hardware since 3.2.5, AIX 4 is in my not-so-humble-opinion the operating system's zenith as a workstation OS. Indeed, this was the PowerOpen era, with common ABI support with Mac OS, and there was even some period of time in which AIX might have been the next Mac OS. For that matter, OS/2 was still a thing on the desktop (as Warp 4) despite Workplace OS's failure, Ultimedia was a major IBM initiative in desktop multimedia, and the Common User Access model was part of CDE too. AIX 4 had multimedia capabilities as well through its own native port of Ultimedia, supporting applications like video capture and desktop video conferencing, and even featured several game ports IBM themselves developed — two for AIX 4.1 (Quake and Abuse) and one later for 4.3 (Quake II). The 4.1 game ports run well on ANS AIX with the Ultimedia libraries installed, though oddly Doom was never one of them. IBM cancelled all this with AIX 5L and never looked back.

ANS "Harpoon" AIX only made two standard releases, 4.1.4.1 and 4.1.5, prior to Gil Amelio cancelling the line in April 1997. However, ANS AIX is almost entirely binary-compatible with regular 4.1 and there is pretty much no reason not to run 4.1.5, so we'll make that our baseline. Although AIX 4.3 was a big jump forward, for our purposes the major difference is support for X11R6 as 4.1 only supports X11R5. Upgrading the X11 libraries is certainly possible but leads to a non-standard configuration, and anyway, the official id Linux port by Dave Taylor hails from 1994 when many X11R5 systems would have still been out there. We would also rather avoid forcing people to install Ultimedia. There shouldn't be anything about basic Doom that would require anything more than the basic operating system we have. The NCommander AIX Doom port is based on Chocolate Doom, taking advantage of SDL 1.2's partial support for AIX. Oddly, the headers for the MIT Shared Memory Extension were reportedly missing on 4.3 despite the X server being fully capable of it, and he ended up cribbing them from a modern copy of Xorg to get it to build. Otherwise, much of his time was spent bringing in other necessary SDL libraries and getting the sound working, neither of which we're going to even attempt. Owing to the ANS' history as a heavily modified Power Macintosh 9500, it thus uses AWACS audio for which no driver was ever written for AIX, and AIX 4.1 only supports built-in audio on IBM hardware. Until that changes or someone™ figures out an alternative, the most audio playback you'll get from Harpoon AIX is the server quacking on beeps (yes, I said quacking, the same as the Mac alert sound).

However, Doom Generic is a better foundation for exotic Doom ports because it assumes very little about the hardware and has straight-up Xlib support, meaning we won't need SDL or even MIT-SHM. It also removes architecture-specific code and is endian-neutral, important because AIX to this day remains big-endian, though this is less of a issue with Doom specifically since it was originally written on big-endian NeXTSTEP 68K and PA-RISC.

We now need to install a toolchain, since Harpoon AIX does not include an xlC license, and I'd be interested to hear from anyone trying to build this with it. Although the venerable AIXPDSLIB archive formerly at UCLA has gone to the great bitbucket in the sky, there are some archives of it around and I've reposted the packages I personally kept for 4.1 and 3.2.5 on the Floodgap gopher server. The most recent compiler AIXPDSLIB had for 4.1 was gcc 2.95.2, though for fun I installed the slightly older egcs 2.91.66, and you will also need GNU make, for which 3.81 is the latest available. These compilers use the on-board assembler and linker. I did not attempt to build a later compiler with this compiler. It may work and you get to try that yourself. Optionally you can also install gdb 5.3, which I did to stomp out some glitches. These packages are all uncompressed and un-tarred from the root directory in place; they don't need to be installed through smit. I recommend symlinking /usr/local/bin/make as /usr/local/bin/gmake so we know which one we're using.

Finally, we'll need a catchy name for our port. Originally it was going to be ANS Doom, but that sounded too much like Anus Doom, which I proffer as a free metal band name and I look forward to going to one of their concerts soon. Eventually I settled on Harpoom, which I felt was an appropriate nod to its history while weird enough to be notable. All of my code is on Github along with pre-built binaries and all development was done on stockholm, my original Apple Network Server 500 that I've owned continuously since 1998, with a 200MHz PowerPC 604e, 1MB of cache, 512MB of parity RAM and a single disk here running a clean install of 4.1.5.

Starting with Doom Generic out of the box, we'll begin with a Makefile to create a basic Doom that I can run over remote X for convenience. (Since the ANS runs big-endian, if you run a recent little-endian desktop as I do with my POWER9 you'll need to start your local X server with +byteswappedclients or a configuration file change, or the connection will fail.) I copied Makefile.freebsd and stripped it down to

CC=gcc
CFLAGS+=-g -O2 -I/usr/local/include
LDFLAGS+=-L/usr/local/lib
CFLAGS+=-DNORMALUNIX
LIBS+=-lm -lc -lX11

I also removed -Wl,-Map,$(OUTPUT).map from the link step in advance because AIX ld will barf on that. gmake understood the Makefile fine but the compile immediately bombed. It's time to get out that clue-by-four and start bashing the compiler with it.

stockholm:/home/spectre/src/doomgeneric/% gmake
[Compiling dummy.c]
In file included from dummy.c:13:
doomtype.h:63: inttypes.h: No such file or directory
gmake: *** [build/dummy.o] Error 1

There is, in fact, no inttypes.h or stdint.h on AIX 4.1.

stockholm:/home/spectre/src/doomgeneric/% find /usr/include -name 'stdint.h' -print
stockholm:/home/spectre/src/doomgeneric/% find /usr/include -name 'inttypes.h' -print
stockholm:/home/spectre/src/doomgeneric/% find /usr/local/include -name 'stdint.h' -print
stockholm:/home/spectre/src/doomgeneric/% find /usr/local/include -name 'inttypes.h' -print

So let's create an stdint.h! We could copy it from somewhere else, but I wanted this to only specify what it needed to. After several false starts, the final draft was

#ifndef __INTTYPES
#define __INTTYPES 1
typedef signed char int8_t;
typedef unsigned char uint8_t;
typedef signed short int16_t;
typedef unsigned short uint16_t;
typedef signed int int32_t;
typedef signed int intptr_t;
typedef unsigned int uint32_t;
typedef unsigned int uintptr_t;
typedef signed long long int64_t;
typedef unsigned long long uint64_t;
#endif

and we include that instead of inttypes.h. Please note this is only valid for 32 bit systems like this one.

stockholm:/home/spectre/src/doomgeneric/% gmake
[...]
[Compiling i_sound.c]
[Compiling i_system.c]
[Compiling i_timer.c]
In file included from i_timer.c:22:
doomgeneric.h:5: stdint.h: No such file or directory
gmake: *** [build/i_timer.o] Error 1

Obviously we'll change that from to "stdint.h".

stockholm:/home/spectre/src/doomgeneric/% gmake
[...]
[Compiling i_video.c]
In file included from i_video.c:41:
/usr/local/lib/gcc-lib/rs6000-ibm-aix4.1.5.0/egcs-2.91.66/include/stdbool.h:9: conflicting types for `false'
doomtype.h:88: previous declaration of `false'
/usr/local/lib/gcc-lib/rs6000-ibm-aix4.1.5.0/egcs-2.91.66/include/stdbool.h:11: conflicting types for `true'
doomtype.h:89: previous declaration of `true'
i_video.c:128: conflicting types for `col_t'
/usr/include/sys/localedef31.h:48: previous declaration of `col_t'
i_video.c: In function `I_InitGraphics':
i_video.c:239: parse error before `extern'
gmake: *** [build/i_video.o] Error 1

doomtype.h has this definition for a boolean:

typedef enum
{
    false       = 0,
    true        = 1,
        undef   = 0xFFFFFFFF
} boolean;

Despite this definition, undef isn't actually used in the codebase anywhere, and if C++ bool is available then it just typedefs it to boolean. But egcs and gcc come with their own definition, here in its entirety:

stockholm:/home/spectre/src/doomgeneric/% more /usr/local/lib/gcc-lib/rs6000-ibm-aix4.1.5.0/egcs-2.91.66/include/stdbool.h
/* stdbool.h for GNU.  */
#ifndef __STDBOOL_H__
#define __STDBOOL_H__   1

/* The type `bool' must promote to `int' or `unsigned int'.  The constants
   `true' and `false' must have the value 0 and 1 respectively.  */
typedef enum
  {
    false = 0,
    true = 1
  } bool;

/* The names `true' and `false' must also be made available as macros.  */
#define false   false
#define true    true

/* Signal that all the definitions are present.  */
#define __bool_true_false_are_defined   1

#endif  /* stdbool.h */

This is almost identical. Since we know we don't really need undef, we comment out the old definition in doomtype.h, #include and just typedef bool boolean like C++.

The col_t is an AIX specific problem that conflicts with AIX locales. Since col_t is only found in i_video.c, we'll just change it in four places to doomcol_t.

The last problem was this bit of code at the end of I_InitGraphics():

    extern void I_InitInput(void);
    I_InitInput();
}

Here we can cheat, being pre-C99, by merely removing the declaration. This is aided by the fact I_InitInput neither passes nor returns anything. The compiler accepted that.

stockholm:/home/spectre/src/doomgeneric/% gmake
[Compiling i_video.c]
[Compiling doomgeneric.c]
[Compiling doomgeneric_xlib.c]
doomgeneric_xlib.c:14: X11/XKBlib.h: No such file or directory
gmake: *** [build/doomgeneric_xlib.o] Error 1

X11R5 does not support the X Keyboard Extension (Xkb). To make the compile go a bit farther I switched out X11/XKBlib.h for X11/keysym.h. We're going to have some missing symbols at link time but we'll deal with that momentarily.

stockholm:/home/spectre/src/doomgeneric/% gmake
[Compiling doomgeneric_xlib.c]
doomgeneric_xlib.c: In function `DG_Init':
doomgeneric_xlib.c:90: parse error before `int'
doomgeneric_xlib.c:94: `attr' undeclared (first use in this function)
doomgeneric_xlib.c:94: (Each undeclared identifier is reported only once
doomgeneric_xlib.c:94: for each function it appears in.)
doomgeneric_xlib.c:98: parse error before `int'
doomgeneric_xlib.c:100: `blackColor' undeclared (first use in this function)
doomgeneric_xlib.c:108: `whiteColor' undeclared (first use in this function)
doomgeneric_xlib.c:124: `depth' undeclared (first use in this function)
gmake: *** [build/doomgeneric_xlib.o] Error 1

DG_Init() is naughty and didn't declare all its variables at the beginning. This version of the compiler can't cope with that and I had to rework the function. Although my revisions compiled, the link failed, as expected:

stockholm:/home/spectre/src/doomgeneric/% gmake
[Compiling doomgeneric_xlib.c]
[Linking doomgeneric]
ld: 0711-317 ERROR: Undefined symbol: .XkbSetDetectableAutoRepeat
ld: 0711-317 ERROR: Undefined symbol: .XkbKeycodeToKeysym
ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information.
collect2: ld returned 8 exit status
gmake: *** [doomgeneric] Error 1
stockholm:/home/spectre/src/doomgeneric/% grep Xkb *
doomgeneric_xlib.c:    XkbSetDetectableAutoRepeat(s_Display, 1, 0);
doomgeneric_xlib.c:                KeySym sym = XkbKeycodeToKeysym(s_Display, e.xkey.keycode, 0, 0);
doomgeneric_xlib.c:                KeySym sym = XkbKeycodeToKeysym(s_Display, e.xkey.keycode, 0, 0);

XkbSetDetectableAutoRepeat tells the keyboard driver to not generate synthetic KeyRelease events for this X client when a key is auto-repeating. X11R5 doesn't have this capability, so the best we can do is XAutoRepeatOff, which still gives us single KeyPress and KeyRelease events but that's because it disables key repeat globally. (A footnote on this later on.) There's not a lot we can do about that, though we can at least add an atexit to ensure the previous keyboard repeat status is restored on quit.

Similarly, there is no exact equivalent for XkbKeycodeToKeysym, though we can sufficiently approximate it for our purposes with XLookupKeysym in both places:

/*
                KeySym sym = XkbKeycodeToKeysym(s_Display, e.xkey.keycode, 0, 0);
*/
                KeySym sym = XLookupKeysym(&e.xkey, 0);

That was enough to link Doom Generic. Nevertheless, with our $DISPLAY properly set and using the shareware WAD, it immediately fails:

                           Doom Generic 0.1
Z_Init: Init zone memory allocation daemon. 
zone memory: 20158008, 600000 allocated for zone
Using . for configuration and saves
V_Init: allocate screens.
M_LoadDefaults: Load system defaults.
saving config in .default.cfg
-iwad not specified, trying a few iwad names
Trying IWAD file:doom2.wad
Trying IWAD file:plutonia.wad
Trying IWAD file:tnt.wad
Trying IWAD file:doom.wad
W_Init: Init WADfiles.
 adding doom.wad
Couldn't realloc lumpinfo

This error comes from this block of code in w_wad.c:

// Increase the size of the lumpinfo[] array to the specified size.
static void ExtendLumpInfo(int newnumlumps)
{
    lumpinfo_t *newlumpinfo;
    unsigned int i;

    newlumpinfo = calloc(newnumlumps, sizeof(lumpinfo_t));

    if (newlumpinfo == NULL)
    {
        I_Error ("Couldn't realloc lumpinfo");
    }

With some debugging printfs, we discover the value of additional lumps we're being asked to allocate is totally bogus:

W_Init: Init WADfiles.
 adding doom.wad
ExtendLumpInfo: -268173312

This nonsense number is almost certainly caused by an unconverted little-endian value. Values in WADs are stored little-endian, even in the native Macintosh port. Doom Generic does have primitives for handling byteswaps, however, so it seems to have incorrectly detected us as little-endian. After some grepping this detection quite logically comes from i_swap.h. As we have intentionally not enabled sound, for some reason (probably an oversight) this file ends up defaulting to little endian:

#else  // FEATURE_SOUND

#define SHORT(x)  ((signed short) (x))
#define LONG(x)   ((signed int) (x))

#define SYS_LITTLE_ENDIAN

#endif /* FEATURE_SOUND */

Ordinarily this would be a great place to use gcc's byteswap intrinsics, buuuuuuuuut (and I was pretty sure this would happen) ...

ld: 0711-317 ERROR: Undefined symbol: .__builtin_bswap32
ld: 0711-317 ERROR: Undefined symbol: .__builtin_bswap16

so we're going to have to write some. Since they've been defined as quasi-functions, I decided to do this as actual inlineable functions with a sprinkling of inline PowerPC assembly. The semantics of static inline here are intended to take advantage of the way gcc of this era handled it.

#else  // FEATURE_SOUND

static inline signed int LONG(signed int x) {
        signed int result;
        __asm__(
                "rotlwi %0, %1, 8\n"
                "rlwimi %0, %1, 24, 0, 7\n"
                "rlwimi %0, %1, 24, 16, 23\n"
                /* note: 64 bit would need an extsw here */

                /* out */ : "=&r" (result)
                /* in  */ : "r"  (x));   
        return result; 
}

static inline signed short SHORT(signed short x) {
        signed short result;   
        __asm__(
                "rlwinm %0, %1, 24, 24, 31\n"
                "rlwinm 0, %1, 8, 16, 23\n"
                "or %0, %0, 0\n" 
                "extsh %0, %0\n"

                /* out */ : "=&r" (result)
                /* in  */ : "r" (x)
                /* temp */: "0");
        return result;
}

#define SYS_BIG_ENDIAN

#endif /* FEATURE_SOUND */

These snippets are very nearly the optimal code sequences, at least if the value is already in a register. If the value was being fetched from memory, you can do the conversion in one step with single instructions (lwbrx or lhbrx), but the way the code is structured we won't know where the value is coming from, so this is the best we can do for now. Atypically, these conversions must be signed. If you miss this detail and only handle the unsigned case, as yours truly did in a first draft, you get weird things like this: Because the sign was not being extended on shorts we converted, but was extended on values we did not, 16-bit values started picking up wacky negative quantities because the most significant byte eventually became all ones and 32-bit PowerPC GPRs are 32 bits, all the time. Properly extending the sign after conversion was enough to fix it. Now on our remote X debugging session, the game comes up and is playable, and not overly slow via the network. (Insert anti-Wayland snark here.) On the console, however (pardon the colours in some of these pictures because this CRT monitor's red gun is iffy), the keys generally worked but the screen was a mess. You can see several things here, notably that the palette is completely wrong, there is an oversized ghost of the Doom logo that appeared when we pressed ESC, and the "pixels" in general are unreasonably swollen. The ANS' video, not used in any other Apple computer, is based on a 1MB Cirrus Logic 54M30. This setup maxes out at 1024x768 and has no provision for anything beyond an 8-bit colour depth. Ignoring the palette for a moment, the X server and the game seem to disagree on the number of bits per pixel: the game apparently thinks 32, but the screen only allows 8.

Going back to the code, there is provision in Doom Generic for an 8-bit colourmap with the compiler define CMAP256. Since this is a compile-time choice, and we want to support both remote X and console X, we'll just make two builds. I rebuilt the executable this time adding -DCMAP256 to the CFLAGS in the Makefile. We now have proper pixels, but the colour (modulo the iffy red gun) is still very, very wrong. X can absolutely support PseudoColor 8-bit visuals, so we must not be creating a colourmap for the window nor updating it, and indeed there is none in the Doom Generic source code. Fortunately, there is in the original O.G. Linux Doom, so I cribbed some code from it. I added a new function DG_Upload_Palette to accept a 256-colour internal palette from the device-independent portion, turn it into an X Colormap, and push it to the X server with XStoreColors. Because the engine changes the entire palette every time (on damage, artifacts, etc.), we must set every colour change flag in the Colormap, which we do and cache on first run just like O.G. Linux Doom did.

#ifdef CMAP256

static XColor colors[256];
static int called = 0;
struct color { /* from i_video.h */
    uint32_t b:8;
    uint32_t g:8;
    uint32_t r:8;
    uint32_t a:8;
};

void DG_Upload_Palette(struct color *dgcolors) /* Cribbed from OG Linux DOOM */
{
        int c, i;

        if (!called)
        {
                called++;
                for(i=0; i<256; i++)
                {
                        colors[i].pixel = i;
                        colors[i].flags = DoRed|DoGreen|DoBlue;
                }
        }

        /* Set X colormap entries */
        for (i=0; i<256; i++) {
                colors[i].red = (dgcolors[i].r << 8) | dgcolors[i].r;
                colors[i].blue = (dgcolors[i].b << 8) | dgcolors[i].b;
                colors[i].green = (dgcolors[i].g << 8) | dgcolors[i].g;
        }

        XStoreColors(s_Display, s_Cmap, colors, 256);
}
#endif

The last step is to tag the Colormap we create to the X window using XSetWindowColormap. Now we have a proper display. The background windows look odd, but their Colormap will be asserted instead when they are the active window. That's just what you have to live with under 256 colours. The same thing happens with the other AIX games, by the way.

Here are some direct grabs from the framebuffer using xwd -icmap -screen -root -out. In general moving around in the game worked fine (remote X sometimes lagged a bit but the console worked well) except for strafing. Doom Generic's Xlib source doesn't check for Alt keys. This makes sense because it's often something window managers trap, and indeed on CDE Alt-Up or Alt-Down will hop to another window if you don't release Alt soon enough.

But we have an alternative on the ANS ADB keyboard: the Command keys. The Command keys on Harpoon CDE appear to do nothing and that's because they map to nothing, not Meta, Super or even Hyper in Harpoon's X server. Instead, when pressed or released each Command key generates an XEvent with an unexpected literal keycode of zero. After some experimentation, it turns out that no other key (tested with a full Apple Extended Keyboard II) on a connected ADB keyboard will generate this keycode. I believe this was most likely an inadvertent bug on Apple's part but I decided to take advantage of it. I don't think it's a good idea to do this if you're running a remote X session and the check is disabled there, but if you run the 256-colour version on the console, you can use the Command keys to strafe instead (Alt works in either version).

Lastly, I added some code to check the default or available visuals so that you can't (easily) run the wrong version in the wrong place and bumped the optimization level to -O3. And that's the game. Here's a video of it on the console, though I swapped in an LCD display so that the CRT flicker won't set you off. This is literally me pointing my Pixel 7 Pro camera at the screen.

However, this port should run just fine on anything else running AIX 4.1, not just the ANS. Sooooo ... let's get out something else running AIX 4.1! Here's an IBM RS/6000 Notebook 860, another RISC ThinkPad-like laptop that isn't, technically, a ThinkPad. You might see this machine in a future entry. This machine is a good test because it also runs 4.1.5 (but IBM vanilla AIX this time) and its GT20-based LCD display supports exactly the same 8-bit colour visual at 1024x768, yet it's a 166MHz PowerPC 603e, slower and with weaker integer performance than the 200MHz 604e we were previously using, plus it only has half the L1 cache and just 256K of L2.

Despite that, though, the game runs credibly. (Sorry about the bad angles but the 860's LCD isn't as good as the one I had attached to the ANS.) The only thing that I did notice was that disabling key repeat didn't work for the left and right keys — if you hold them down, they turn, stop, and then turn slower — while other keys seem normal. There is no option to change this on a per-key basis in CDE (remember, no Xkb) and near as I can determine the ANS and the 860 are configured the same. There may be a difference in the X server which I'll have to figure out later. Our target system remains the Network Server but it's nice to see it running other places.

In any event, have at it. Now your ANS has something new to do. The source code and precompiled builds both for 24-bit and 8-bit colour are available on Github. Like Doom Generic and the original Doom, Harpoom is released under the GNU General Public License v2.