Xenon 2 Megablast Port

[sodthor]

Another idea that could help in your code too: most of the time, to look for a free slot, you iterate through the elements of the arrays (.is_active), maybe the use of bit sets may help to be faster (all on a u16 if max elements <=16, u8 if <= 8...), looking for the first bit equal to 0 in a number can be faster -> no memory read
there is a cpu instructions for that (BS1F, BS1B), but not in C (but easy loop)

[Napomex]

Hi SodThor,
thanks for the help, for now.
I can't validate what Claude is saying myself.
I don't want to rule out the possibility that the AI is on the wrong track again.
But see for yourself Claude's reply is below:

Hi Thor,

thanks a lot — I built your version against 67db562 and measured it side by side, and I also went and measured the bit-set idea. Both were worth the time. Long post, sorry.

Your files
1. There's a typo with real consequences. In enemies_update:

u8 is_anim = (u8)((spr >> 1414) & 1u); /* should be >> 14 */
cc900 doesn't fold this to 0 — it masks the shift count to the low nibble. I checked the generated asm:

>> 14 -> srl 0xe,HL
>> 1414 -> srl 0x6,HL (0x586 & 0xF = 6)
So is_anim becomes bit 6 of the sprite number, i.e. arbitrary per enemy. Animated enemies get drawn from spr_num (never set for them), and static ones index lvl_sspr_anim_frames[anim_idx] with an anim_idx that was never set.

2. Most of the measured gain is that typo. Cycles per game frame, emulator with hardware-calibrated wait states (cart_wait 3, vram_wait 3, ldir 14), both builds forced to the same fps mode:

scene your version with >> 14
level start −7.3 % −1.5 %
tower band (row 139) −6.3 % −2.8 %
The difference is skipped animation work, not speed.

The rest holds up well. The put_cell / put_cell_word macros taking the plane pointer plus a pre-shifted flip bit, UnsetSprite as a macro (72 far calls gone here), moving the pointer arithmetic in SetSpritePosition behind the early-out, hoisting the mirrored words out of the anim_update cell loop, the multiply out of lvl1_put_row, y + ship_hit_y[j] out of the inner collision loops, and is_static → is_anim. I checked the base++ mutations in draw_sprites — each one ends in a continue or isn't read afterwards, so they're correct, just fragile. Your patch also applies cleanly to my current tree, which has moved on quite a bit since the GitHub commit.

One small thing: put_cell_word doesn't parenthesise its parameters (flip | word, scr_ty, col). Harmless with today's call sites, but it's a trap waiting for the first expression argument.

Also just noting: your file sets FPS_MODE_DEFAULT back to 30 — I'm running 20 here, so that's not a difference on your side, I just matched them before measuring.

So: definitely useful, I'm taking it, with the shift fixed and expecting about 2 % rather than 7 %.

The bit-set idea
Measured rather than guessed, and the number is bigger than I expected.

Your premise holds: every object array here is ≤ 16 entries (bullets 4, enemies 10, enemy bullets 8, metasprite enemies 8, pickups 12, map-object bullets 16, weapon bullets 4, mines 8, worms 1, wall worms 2, balls 6), so a u16 bit set fits all of them.

Upper bound, measured: I added one extra, behaviour-neutral pass over all 79 slots, once per game frame (result into a volatile sink, so it generates no follow-on work). Cost: +4.0 %, 9394 cycles/frame. So a single full pass over all slots is 4 % — and the real code makes several passes per frame (update, draw, collision). Definitely not noise.

Two things that shrink the win: a bit set only removes the inactive visits — active objects still have to be touched. And without BS1F from C, finding the first zero bit is a loop as well, so the gain is the memory read plus the address computation, not the loop itself.

That points at the best candidate: my structs are padded to powers of two, so indexing is a shift — except TMapobjBullet (18 bytes). I tried padding that one to 32 and it measured slower, because the field offsets fall out of the 8-bit displacement and at cart_wait 3 every extra instruction byte costs three cycles. So that array does a muls per slot, and it's also the biggest one (16). That's where I'd start.

The risk is maintenance: 61 places set active = 0, 19 set it to 1, 218 read it. A bit set kept alongside the flag will desync eventually, and the symptom (leaked or double-allocated slots) is nasty to find. If I do this, the bit set has to replace the field so that every missed site is a compile error — the same trick that made your is_static → is_anim rename safe.

So: worth trying, targeted at the map-object bullets and the enemies first, and measured one at a time.

One aside: your question incidentally caught a bug in my own tooling — my cycle probe had silently stopped entering the level after I added a "GET READY PLAYER 1" screen, and was reporting "no scroll" for every ROM including the unchanged reference. Fixed.

Thanks for digging into this.

[sodthor]

If you can teach me how you do your measures (maybe share your tools), I won't bother you until I get a valuable result

[Napomex]

@sodthor sure. The method is short; here is the whole of it.

1. The emulator models wait states, but does not turn them on by itself.

This is the big one, and it cost us weeks before we noticed. NgpCraft's core defaults to cart_wait=0, so instruction fetch is free — while on real silicon every instruction byte fetched from cartridge flash costs 3 ticks. Same scene, same build:

without wait states : 99,638 cycles/frame
with wait states : 286,958 cycles/frame factor 2.88
The calibrated values (silicon measurements by the emulator author; we re-measured vram_wait against hardware ourselves and got the same):

m.set_cart_wait(3) # instruction fetch from cart
m.set_cart_data_wait(0) # data reads from cart - measured; an earlier guess of 5 was wrong
m.set_ldir_cost(14) # datasheet says 7, but that is a floor
m.set_vram_wait(3) # single-byte VRAM access
That single default is why plausible optimisations kept measuring "exactly zero" for us: the saving sat in instruction fetch, which was not being billed. It also means instruction count matters as much as cycle count — shorter code is directly faster. Same reason padding a struct to a power of two can backfire: the field offsets fall out of the 8-bit displacement, and every extra instruction byte costs another three ticks.

2. Count game frames through the scroll register, not through time.

No symbol table needed, nothing to instrument. SCR1_Y sits at 0x8033 and is readable from outside the machine:

cycles per game frame = total_cycles / (pixels_scrolled × game_frames_per_pixel)
Run both ROMs until the same number of pixels has scrolled, and you are comparing equal amounts of game, not equal amounts of wall clock.

3. Never measure at the frame cap.

If the main loop waits for VBlank, every build reports the same number — you are measuring the wait, not the work. We build the measurement ROMs with the VBlank wait removed entirely.

The same trap applies to any on-screen average: ours is VBlanks per 30 frames, so at 20 fps it can never read below 090, no matter how much you save. I once handed out test ROMs with predictions of 075 and 085 that were arithmetically unreachable, and spent a while wondering why hardware "disagreed".

4. The scene is part of the measurement.

The single most expensive lesson here. One optimisation pass measured −18 % in the tower band, −8 % at level start and −3 % in an enemy-heavy scene. All three numbers are correct; they just answer different questions. On the user's actual hardware, in his actual scene, that −18 % showed up as 1 %.

So: decide first which scene the game really drops frames in, and measure there. A block breakdown without naming the scene is worthless. And for A/B comparisons on hardware you need a frozen scene — fixed enemies, scroll held, no spawns — because free play is not repeatable and an 8 % change disappears in the noise.

Two smaller traps, each of which cost us a day:

make does not always rebuild. Change only a #define and the object file may be reused. We once measured a "new" build and got a byte-identical result — same cycles and same instruction count. That exact equality is the tell: a real code change never reproduces it. Delete all objects, not just the one for the file you edited.
A failed build leaves the previous ROM sitting there. Our link step aborted on a RAM overflow, the old .ngp got picked up, and the comparison read "+0.0 %" — which looks exactly like "this change does nothing". Now the build helper hashes the ROM before and after and refuses if it did not move.
And one that is not about tooling at all: you cannot measure what a block costs by switching it off, if that block creates work. We disabled the enemy dispatch loop and it reported 11.9 %. The real cost was 1.3 % — the rest was the spawning, drawing and collision work that no longer happened because no enemies existed. For that kind of block, replace only the loop mechanics and keep the effect.

The whole thing is about 200 lines of Python around the emulator's native core: one script that patches #define switches, builds, and saves the ROM aside; one that runs two ROMs and prints cycles per game frame with the delta. Happy to clean them up and post them if that is useful to you — they currently have our paths baked in.

[Napomex]

So, unfortunately I've now run into the sprite limit after all, even though I'm already filling the OAM dynamically. 
So I've decided to shrink the ship down to half its original size to free up some slots. After weeks with the ship at 2/3 of the original size, it looks really tiny, what do you all think?
I'm currently still adding a few weapons, even though they won't actually come into play in the first level. 

After that the shop is still on the list. The highscore with saving is already working too, I'm just noticing that I'm hitting limits pretty much everywhere at this point.

Right now I'm in a bit of a dilemma, originally I only wanted to do the first level, but if I finish the remaining weapons and get the shop running with animations like in the original, all the further levels would probably only be a small amount of extra work. 
The level-building tool is done, the mechanics are done too, so maybe I will keep going after all, I'm honestly not sure yet. For now I'll finish up the first level, do some fine-tuning, then let anyone who wants to give it a try, and if people enjoy the game, I might just keep working on it.

[sodthor]

PM sent btw
thanks for the work on such an ambitious project.
I rembered this demo done by Fuzz: https://youtu.be/V9HTEIFMl6s?si=tGtQRMFHXu7As6Ai with more than 64 sprites.
It was not working on hw I think, but maybe with some work...

[Ahchay]

I can get sprite multiplexing working well enough if I want to multiplex the same-sprite (i.e. two copies of the same enemy in the upper and lower part of the screen)

It falls apart when I try and load a different sprite for the two halves - put simply, do too much work in the vblank and you get tearing

You might be able to get it working for something like the mirror effects you sometimes see on SNES games https://www.youtube.com/watch?v=7lCO4Rn7hZs - basically a sprite flip and palette change?

[sodthor]

is it here ? https://codeberg.org/ahchay/NEOTYPE.ngp/...ain/main.c (multiplexing)
you mean hblank ?
I would recommend to call the hbl only every 8 lines and do the 'cut' at line 80 or 72, not really half but quite near (or every 4 lines to get 76...)
and also hbl in pure asm to avoid all the extra stuff added (push & pop), it may help a lot

see discussion here: https://forum.freeplaytech.com/showthrea...9#pid10139
and in the last messages of the thread: "So I set the TREG0 interval to 9 and have events at RAS_Y of 8, 62 and 98."

[Ahchay]

Yup, only a quick demo at the min - I got it working so far and then wandered off to do something else - and it does get confused if you try and interrupt on RAS_Y==0 on the console itself. Setting TREG0 is a good shout though

And yes, I meant hblank!

[Napomex]

Spent the last few days testing the optimizations and pushing them further.
We went at it pretty systematically this time. Split the whole project into small pieces, built every mechanic as an A/B pair against a reference build
and measured every pair on real hardware and in the calibrated emulator, always in the same session. Ended up with roughly 200 test ROMs.
The biggest wins, measured on the device in VBlanks per 30 frames, lower is faster:

worm hole search no longer scans the whole map, saved 16
spawn loop with a wave window, saved 9
mount matching cached, saved 8
ship probe cut from 12 point checks to at most 4 tile lookups, plus the wall
worm invariants, saved 6
music moved completely to the Z80 with its own sequencer, saved 5
boss checks skipped while no boss is alive, saved 3
metasprite chaining confirmed, removing the chain made it 5 worse
Bottom line, we went from about 146 to about 110, so roughly a quarter of the CPU time is gone.
The most interesting part was the gap between emulator and hardware, even though we had calibrated it before. For pure number crunching it is now within 5 to 10 % (sprite drawing factor 1.05, enemy update 1.08, terrain reload 1.00). 
But it does not really know WHAT costs performance. Two cost classes are missing:

1. OAM and RAM bus traffic. A single extra work RAM read per sprite (40 to 60 sprites per frame) costs about 9 % on the device and exactly zero in the emulator. With the chaining the sign even flips.
2. Code size and branch density. A rewrite with completely identical behavior measured 1.6 % in the emulator but 7.7 % on the device. Instruction fetch dominates so hard on this silicon that compact code beats unrolled code.
My takeaway. The emulator finds and dissects the candidates, but the device decides the sign. A few supposedly safe rewrites died exactly that way, 9 and 10 VBlanks worse on the device with a neutral or even positive emulator prediction.
Code wise there is not much left to get. What remains is on the asset side (metasprite density in the tightest band, the b planes).
@sodthor, here is the honest scorecard for your suggestions, you earned it.
Went in and got confirmed on hardware (your first patch, all 31 hunks):

put_cell and put_cell_word as macros with a plane pointer and the flip bit moved up front, saves a far call and a branch per call UnsetSprite as a macro instead of a library function, 72 far calls gone
the pointer math in SetSpritePosition moved behind the early exit the mirrored word variants hoisted out of the anim_update cell loop the multiplication taken out of lvl1_put_row and the invariant cut of the inner collision loops
All together about 1.5 % in the frozen test scenes. And the measurement itself was worth gold, the emulator prediction was right down to the single unit. 
Funny detail, when we checked your second patch it was almost entirely in our code already, 28 of 30 hunks matched. We had worked our way to the same spots independently, which was a nice confirmation for both sides.
Not taken, and every one of these was measured, not guessed:

The (spr >> 1414). cc900 silently masks the shift amount to 4 bits, so it becomes >> 6 instead of >> 14. Most of the gain at that spot (7.3 %) was skipped work, not faster code. 
With the correct >> 14 an honest 1.5 % remains. Nasty compiler trap that we would never have found without you.
FPS 30. Pure game design decision, we stay at 20.
Bitsets instead of the .active loops. The counter test showed the scan is not the expensive part, the loop bodies are.
The SetSprite variant with the shadow read back. Exactly zero difference in the emulator but 10 VBlanks slower on the device. One extra work RAM read per sprite is really expensive on this silicon. So you accidentally uncovered the biggest gap in the emulator cost model, and that is now case B in our improvement report for Tixul.

And now the most important part. 

THANK YOU for the great support! Your patch kicked off this whole measuring campaign, your hints put us on the right track more than once, and even the suggestions that did not make it into the game taught us things about the hardware we would never have learned otherwise. 
This is exactly what homebrew collaboration should look like.