andrewbroad

I would say Geoff Mode III. The Patch Vector system is implemented differently for a start. (There are both Room Setup and Main Loop patches, each of which only uses a single offset byte in the room definitions.)

I just manually took the algorithm's new route through the Bug-Byte version of The Warehouse (i.e. top-right item collected last), and I managed to match the previous best score for the BB version of 1951.  But I don't think the SP version's best score of 1952 is possible in the BB edition.  The threshers are wider than the spinning triangle, so you have to wait for a single time-frame for the right-most thresher to get out of the way before making the jump towards the portal.

Au contraire!
 

Hi all,
I've started to work on building tools to generate a walkthrough of JSW (no estimations on timeline yet, and whether I have enough enthusiasm to finish yet; I hope so, although it feels a larger project than MM 🙂 ).
I have a few questions on how to interpret the memory state (I have assumptions for most of those but it would be nice to confirm it).
1. There are 1-byte 0x85CF "Willy's Y location" and 2-byte 0x85D3 "Willy's attribute location in a buffer" variables is the game.
At the beginning of the frame (when PC=0x89AD), is it always true, that memory[0x85D3]=0x5c00+32*(memory[0x85CF]/16)+x_offset?
In other word, can it happen that they go out of sync (on the rope, on the ramp, during the jump, during room transition etc)?
2. Is it true, that [at the beginning of the frame (when PC=0x89AD)], I can ignore/write garbage to 0x85D5 "Jump animation counter" if 0x85D1 "Airborne status indicator" is not 1?
3. To save the resources and not try to press keys when Willy is jumping/falling, in Manic Miner I used the following logic:
a) If before the frame, "Airborne status indicator" is 0, try all input combinations, else only try "no keys pressed"
b) If _AFTER_ the frame, "Airborne status indicator" is 0, also try all other combinations for the previous frame (if didn't try already).
Questions (both for MM where it seemed to work, and in JSW where I hope it to work):
- Is it enough? (e.g. are no special cases like falling onto the edge of conveyor belt etc)
- Maybe there's simpler logic?
4. It looks like "Willy's Y location" is rounded to the multiple of 16 if Willy is on the ramp. How exactly the logic that draws Willy goes then?
"If Willy is in the same cell as ramp and not jumping/falling, move him up depending by the 0x85D2 "Willy's animation frame" value, but if Willy is jumping above ramp, 0x85CF "Willy's Y location" contains true Y position" -- is that understanding correct?
Thanks!

There's a subroutine at #8E9C which is called to automatically ensure that attribute and pixel y-coordinates stay in sync when Willy is jumping/falling/on a rope.
https://skoolkid.github.io/jetsetwilly/asm/8DD3.html#8e9c
In up/down room transition, Willy's attribute coordinates and pixel y-coordinate are manually adjusted, one after the other.  Ditto for when Willy moves along from one ramp cell to the next (though visually Willy is drawn on a ramp as Norman describes, unless he's jumping through the ramp).

Further to the topic of orientation of Willy upon game start. This automatically causes an auto play problem. If willy needs to turn around at game start, you have wasted a whole frame. The start orientation if collecting the bath tap first has a change of one frame between the differences in direction. For playing the whole game one frame will not be detectable unless playing to 1/100 sec style timings. Once the initial room has been played then Willies state from that room onwards is set.  However that initial one state change can/could cause problems. That's  #85d0

The value in #85d2 the animation frame adds another problem

This does pose the question. Do you force the initial state as being the state that is in operation when the game is loaded. 
--------------------------------------------------------------------
Stairs
I glanced at your questions and do wonder why you are concerned about the pixel shift of Willy when on a stair.

When on a stair the game treats him as standing on a platform. If you remove the code for stair displacement. Willy walks up the stairs as a series of steps. He is treated as being on character/tile boundaries and NEVER treated as being displaced from the Character/tile boundary. (jumping is another story). On a stair he is visually drawn shifted to appear as in he is moving diagonally. The game code whilst on a stair  treats Willy as stepping up the stairs one character at a time.  

In a nutshell. If willy is on a platform- The code checks the direction of the stairs For each direction of stair we check the block directly below his feet. For a stair going from left to right upwards /  we check the block beneath his right foot for a stair tile. For a stair going the other way we check the block under his left foot for the stair tile.

If the stair tile/block is present we displace Willy from the vertical (only on the drawing of Willy and this displacement is not saved anywhere)

For stairs /   we use the animation frame and add a displacement of 6/4/2/0 (pixels) for each animation stage
for stairs sloping the other way we add 0/2/4/6(pixels) for each animation stage. 

As stated Willy never code wise moves from the fixed character/tile boundaries- the displacement is an optical trick to make him look like he walks up and down the stairs. 
==========================================================================================================
Addendum the pixel offset is doubled for the YTABLE offset - to move willy down the screen 2 pixels we add 4 to his YTABLE position, since each entry in the YTABLE is 2 bytes.

Good spot.
The way the test for ropes comes first in the routine, it's almost as if Matthew had this cheat in mind!

The simplest modification I can come up with is just add a 3 byte modification such as:-
In decimal
37314    LD A,(IX+0)    Pick up the first byte of the current entity's buffer
37317    CP 255    Have we already dealt with every entity?
37319    RET Z    Return if so
37320    AND 7    Keep only bits 0-2 (which determine the type of entity)
37322    JP Z,37811    Jump to consider the next entity buffer if this one is not being used
37325    CP 3    Is this a rope?
37327    JP Z,37540    Jump if so
37330    CP 4    Is this an arrow?    <<<<<<<<<<<<<<<  mod to JP 37811
37332    JR Z,37431    Jump if so
In hex
91C2    LD A,(IX+$00)    Pick up the first byte of the current entity's buffer
91C5    CP $FF    Have we already dealt with every entity?
91C7    RET Z    Return if so
91C8    AND $07    Keep only bits 0-2 (which determine the type of entity)
91CA    JP Z,$93B3    Jump to consider the next entity buffer if this one is not being used
91CD    CP $03    Is this a rope?
91CF    JP Z,$92A4    Jump if so
91D2    CP $04    Is this an arrow? <<<<<<<<<<<<<<<  mod to JP $93B3
91D4    JR Z,$9237    Jump if so
 
This kills all the sprites but leaves the ropes alone. So you can wander ALL the rooms including Esmeralda
 

Another little-known fact about JSW - Willy can be detected as having reached the toilet during the toilet-run if he is eight character-rows above the toilet, in the upper half of the room.  (If you extend the ramp in The Bathroom down to meet the floor, and put a hole in the upper platform above the toilet, then start the toilet-run, watch as he runs up the ramp, drops down the hole, and then disappears mid-descent rather than falling all the way into the toilet!)
But I've gone way off topic now! 😁

Quantum Tunnelling

Willy passes through objects.

 

Great stuff @crem!
 
Could you check the level 17 rzx? When I play it back I only see Willy dying twice.
 
Five hints that @jetsetdanny is actually human? Or five hints that the jsd AI is so advanced to know that it'd better underperform a few times to prove its humanity? 🤔

In a way, Norman's data stream is similar to a manual method I sometime use when approaching a pixel-perfect jump. e.g. travelling along the conveyor in Central Cavern (where the conveyor is doing the work in terms of horizontal movement), in order to ensure that I jump at the last possible moment (so as to land on the 'intermediate' platform, bypassing the one which occupies the same character row as the conveyor itself), I do the following:
Keep the in-game Pause key (usually 'A') pressed, and make rapid stabs at a non-functional key (such as '1') to make WIlly progress one frame at a time along the conveyor.  That way the game pauses in between each frame of movement (if I'm still pressing 'A' throughout).  Then when I can see that Willy is teetering on the brink on the conveyor, I hit the jump key, and only then let do I go of the pause key.

Scripted Movement.

I altered the keyboard input routine to accept input from a data stream. The data stream is simply a file that I write, with values for the various inputs ( left, right, jump) The room is played and the data is taken from the data stream one item at a time, which simulates the keyboard input. I simply play the room and watch what happens, and add or delete the data to make the jumps at the exact timing I want. After I have perfected a particular jump or movement. I start adding more data to simulate movement to the next point of interest. The data streams are simply optimised movement data from myself. I played (typed data) for the first two rooms, just to establish what my optimised play would do. 

This method is Not going to find the best route. It will play a route that is picked and adjust the jumps and walks to one frame accuracy.

Part of one data file (generated by visual play - type data and watch what happens)
    DB    0,0, 0,0,0, 0,0,0,0,0,0,0, 0,0,0,0,0
    db    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
    db    1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1;,1,1;,1,1,1,1,1,1    ;START FALL
    DB    0,0,0
    DB    1,1,1,1,1,1,1                  ;WALK
    DB    5
    db    0,0, 0,0,0, 0,0,0,0,0,0,0, 0,0,0,0,0    ;jump collect
    db    0,0,0,0,0,0,0,0,0,0,0,0,0,0         ;floor collapse
    db    2
    db    6                                       ;jump out of hole
    db    0,0, 0,0,0, 0,0,0,0,0,0,0
    DB    2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2        ;WALK TO FALL
    DB    0,0,0,0,0,0,0,0,0
 

The first two  cavern played by scripted movement.

one cavern per script
Score 1724 (cavern 1) and 1975 (cavern 2)

Not carrying on- interesting one night exercise.
--------------------
Watching cavern 7's erratic generated movement. I decided to do my own. The route taken by my script is a bit more relaxed. Still gets the same score.

This has shown me not to do any more, it just is not what I want to do with my time.
 
 
 
sc1_1724.tap sc2_1975.tap
sc7_1824.tap

That route in Wacky Amoebatrons is a shocker! 😉 It looks on like on several occasions WIlly has made proper contact with baddie sprites to me, he changes colour due to proximity. Further, I suppose one huge advantage that the AI has is that it can make perfect jumps every time off the very end of ledges without having to worry about keypresses being correctly read.

Sorry for posting this level out of order, but I think it's an improvement of the best known score by almost three seconds (interestingly, with the "only start moving from the second frame" bug, it was actually almost 3 seconds slower than the best known score, due to having to wait for the blue amoebatron in the very beginning).
Level 9 ("Wacky Amoebatrons")
Score: 1376

lvl9.mp4 lvl9_20210312-0255.rzx

Way ahead of you on this one:
 
 

To take this a bit further, the following three commands in the JSW128 Cheat routine could be edited to change the half-rows of keys that are picked up:
At #C428, there is LD BC, #FBFE (01 FE FB in opcode)
At #C446, you will find LD B, #DF (06 DF)
At #C45E, we have LD B, #FD (06 FD)
Changing the value that is loaded up to B in one or more of the above commands will change the half-row(s) from which keypresses are detected.  Doing that in conjunction with editing the data at #C480-#C49A could allow you to have a 9-letter Cheat password that draws characters from any three half-rows of keys.
e.g. POKE #C45F, #FE changes the existing password to ZUPERUZER (loading B=#FE allows you to access the half-row SHIFT-V, and in this case there is no need to edit the table at #C480-9A, because 'Z' occupies the same position as 'S' in their respective half-rows - pressing them resets Bit 1 in both cases).
Note that if you're using a key like 'Z' as part of your password, which also functions as a Jump key, then you have to wait until Willy has landed again after pressing it before proceeding with pressing the next character in the password sequence - I've just tried it though and it does work!
There is a very handy table below (from the following link), which gives a full list of all the ports and bits that correspond to each key on the Spectrum:
http://www.breakintoprogram.co.uk/computers/zx-spectrum/keyboard
 

I do recall observing that glitch a while ago (and coming up with an explanation how it works in terms of the code - I think the Main Loop gets stuck in a, er, loop!)
You can easily get out of it though, without resetting the game, by just toggling Invincibility Mode back off again.🙂

Thanks for sharing this, crem, it's fascinating! 👍
I meant to make exactly the same comments as Ian made above (great minds...! 🙂), drawing your attention to the highscore challenge scores and the Solar Power Generator, where you will have to optimise the method for the remaining air, as the solar beam is a huge factor there.
I am also wondering about JSW games, as the possible routes will multiply the possible options the algorithm will have to analyse. The original "JSW" may be "relatively" simple in this respect, but if you ever wanted to carry out a similar analysis for other JSW games that have been released so far (a complete list is here), you will find some real behemoths among them. Take sendy (Alex Cornhill)'s "where's woody?" for example. 256 edited rooms (although the player does not have to visit all of them to complete the game - that's another challenge for the algorithm, to determine which rooms can be skipped safely), non-linear, hundreds of possible room exits altogether, so I would think thousands and thousands of possible routes to take (or even more - it will grow exponentially, won't it?). I don't know if you have enough processing power to analyse that one 🤪. But it would certainly be interesting to know how much my best in-game completion time (achieved using the Rollback feature and what I decided, after a careful analysis, was the most efficient route) of 2:10 pm can be improved...
Meanwhile, I am looking forward to your scores in the next MM caverns - it's very interesting, thanks again! 🙂

There was actually a 20-page long thread exactly on this topic on chess engine developers forum, whether aggressive variants pruning based on hauristics makes the algorithm non-brute-force. It's just a question of terminology, and most of people seem to agree that brute force with pruning is still brute force, as opposed to novel neural network based algorithms.
For Manic Miner, as I've said, I use score-based pruning. Just keep N results with the best score for every frame. The score is pretty simple:
Every collected item adds 1.0 to the score. If Willy is jumping or falling, that subtracts ~0.001 from the score -- not sure this is necessary. If the position of Willy (ignoring the rest of the level state) wasn't seen earlier, add 0.9 to the score. In the case of duplicate Willy positions within N frames in the batch, penalize the second one with -0.01, third with -0.02 and so on.
Those two latest rules help with diversity of positions. Otherwise Willy would stuck in lots of copies of the same position with different world state (e.g. all possible combinations of crumbling platforms) If the same Willy's position was seen in previous frames, also penalize that with -0.003 per frame. It works pretty well, although there are issues. For example, when Willy stays on a crumbling platform, he doesn't change his position for 8 frames (he only starts to fall when the platform fully collapsed), so this is penalized and in general Willy doesn't like to use them.
Also as you could see, it doesn't like to wait, instead it will do useless back and forth movements and so on. Speedrunners actually liked similar behaviour in Prince of Persia, it allows them to time the strat correctly instead of just waiting for the right moment.

I have finally managed to crack how to turn on "the other" cheat mode (i.e. other than the Writetyper cheat) in JSW128 and JSW64 games, at least partly 😁.   I would like to thank Ian for his persistent encouragement to have another go at this secret, and for his tips expressed earlier in this thread. However, I should note that I did not make use of them - I couldn't arrive at the solution on the basis of the code. In my "final assault" I cracked the mystery by a mixture of the facts I had figured out before - that the cheat mode indicator is at #C400, the value which turns it on is #09 and that the first letter of the password is "s", which changes the value of #C400 from #00 to #01 - and brute force: I typed in letter after letter to see which one will change #C400 from #01 to #02, then from #02 to #03 and so on. In fact, when I had six characters of the word I am going to reveal below, I guessed the remaining three, so the brute force was complemented by an educated guess 🤪.   So here it is:   The cheat mode can be activated by typing "superuser" when you are at the floor in room 046 (which in "Jet Set Willy 128K" and "Jet Set Willy 64K" is called "Root"; in other games using this game engine it will have a different name). After a brief border effect, the cheat-mode tune kicks in instead of the regular one and a hash symbol appears to the right of the last remaining life (if both the Writetyper cheat and the other cheat mode are activated, the Writetyper foot is next to the last remaining life, and the other cheat mode symbol is to its right).   At this moment the player has acquired the ability to carry out what John Elliott calls "fast inter-room teleportation": they can move to the room to the left by pressing "z", to the room to the right by pressing "v", to the room above by pressing "c" and to the room below by pressing "x".   This tip should also work, in principle, in all of the other games using, respectively, the so-called JSW128 game engine (of which "Jet Set Willy 128K" was the first one) and the so-called JSW64 game engine (of which "Jet Set Willy 64" was the first one). So far, altogether, there are 18 gamma-released games using the JSW128 game engine (here's a complete list) and 24 games using the JSW64 game engine (here's a complete list).   When I say the cheat mode should work "in principle", I am referring to the fact that whether or not it can be activated without using POKEs depends on the design of each game. If there is no room 046 at all (because it hasn't been edited), or the bottom of this room is not accessible for the player to type "superuser" while over there, the cheat cannot be activated in a normal way. I did not try to check all of these games to see if the cheat can be activated in them without POKing them or not.   Should it not be possible to activate the cheat by typing "superuser" at the floor in room 046, it can be activated by applying POKE 50176,9. It should be noted, however, that if it is done while playing the game, the cheat will be activated (along with the hash symbol), but the in-game music will remain the regular tune. To be able to hear the cheat-mode in-game tune, one would have to either apply POKE 50176,9 on the title screen, before starting the game, or, if it is applied while playing the game, the player would have to lose all their lives (or reset the game by pressing Shift and Space together) and restart it; then the cheat-mode in-game tune will kick in.   Both Writetyper and the other cheat mode stay on after the game is over. They are not reset at the start of the next game. Therefore, if someone POKEs in 50176,9 before starting the game and starts with the cheat-mode in-game tune on, the only way to turn it off (i.e. hear the regular in-game tune) would be to POKE 50176,0 (to turn the cheat mode off) and reset the game or restart it after losing all lives, or reload the game.   Once the cheat mode is turned on, I don't think it's possible to turn it off by any built-in mechanism, without POKing the game. It's the same as with the Writetyper cheat - once you have activated the cheat, if you want to deactivate it, you either need to reload the game (to start playing again without the cheat activated) or to POKE the game to turn the cheat off.   A thing that should also be noted is that, for some reason, the cheat-mode in-game tune is *much* louder than the regular "Jet Set Willy 128K" in-game tune. I'm not sure why it is like this, but the difference in the level of loudness is very pronounced, at least to my ear. The difference is not as big in "Jet Set Willy 64K" (which has a different regular in-game tune than "Jet Set Willy 128K").   So now I hope that Ian will reveal how to toggle on/off the other two features: invincibility and antigravity 😉. Please, go ahead, Ian, and let us know; it's long overdue 🙂.  

Thanks for the info and the loader, Andy! 🙂
For me, personally, the official Software Projects POKEs and the colour code bypass are the only things I would apply as fixes to game variants such as "JSW Mono". These are the things necessary to launch the game and to complete it. Nothing else is necessary, so I don't take it into consideration. I know, it's just a matter of opinion and personal choice, and it's different with variants like your 2015 Bug Fixed Edition, where the whole idea was to correct various issues considered to be bugs, not only the critical ones.

A small twist on the existing 'mono' option. If applied manually its as follows, the 'hexpat' executable file does not function on x864 machines :
ORG 35339
LD A , 71
JR 35354
We can have a little fun with this by picking up the border value instead:
ORG 35339
LD A , (32990)
JR 35354
    
Potential here perhaps to use an existing unused byte value per room to individually set a room colour independently of border colour, however the above works OK.
I did toy with a small release of this as the existing release of JSW Mono does not have any bug fixes applied at all.
If your screen flash code is missing ( ! ) then the whole lot at 35354 should read (it then continues onwards anyway into valid code)
ORG 35354
LD HL , 23552
LD DE , 23553
LD BC , 511
LD (HL) , A
LDIR
^ That should not be necessary in most instances of JSW1, its possible some third party games may have modified or used it however.
Manual POKE's for the above "multi mono" code:
POKE 35339 , 058
POKE 35340 , 222
POKE 35341 , 128
POKE 35342 , 024
POKE 35343 , 010
Pic:

 

Thanks again, everyone, for all of your comments! 👍 To answer RuffledBricks's question - no, I'm not on Discord.
I guess that the time difference has been explained satisfactorily. I did not realise turning the music off made such a difference, plus indeed the initial comparison was of a with-lives walkthrough with a life-lost walkthrough, so there must have been a difference.
I find it amazing how deeply members of the speedrunning community go into detail about how to improve the results, and at the level of perfection RuffledBricks and others must have when playing the game. It's another kind of (wonderful) craziness, a bit like we do here sometimes when we discuss some extremely tiny details of game projects, or pore over the code to find a one-byte saving, and things like that.
The intriguing thing is where the optimisation of walkthroughs will stop. I think that we are now very close to perfect routes and executions both in JSW and MM, possible improvements are minimal, and in many rooms simply not possible any more.
Is Matthew Smith aware of the existence of the speedrunning community and your achievements? I think it should thrill him to know how much time people spend on his two games and how much effort they put into activities related to them - now almost 40 years after he created them...