RuffledBricks

Some updates on further levels (tl;dr no success yet):
For level 4 ("Uranium"), the computation was completed, but for some reason Fuse fails to open the resulting .rzx file. Will look why is it so when I have time (hopefully today in the evening). I also have keypress information per frame, so if it's the problem with rzx writing and the rest worked fine, there will be no need to redo the calculation.
Level 5 ("Eugene") has the same problem with opening .rzx, but also it didn't find the win. When I woke up today and checked the results, all 35000 states had Eugene already sitting on the top of the portal, making win impossible. I'll restart this level with adjusted scoring (will make it a loss if Eugene sits on the portal).
In the morning I started Level 6 ("Pacman Processing Plant"), Level 7 ("VAT"), Level 8 ("Kong v1") and Level 9 ("Amoebatrons v1"). For levels 2-5, the search with "width" of 35000 states took 6 hours to complete, so I started Levels 6-9 with 50000 to finish by the end of the day.
Level 7 ("The Vat") finished after 7 frames with "no solution found" status. Clearly some bug on my side, will take a look later. Meanwhile, started Level 11 ("Telephones") instead (and later realized that I missed Level 10 "Endorean Forest").

Another fundamental difference in cultural mindset on display here - I enjoy the site of lots of air being counted down at the end of a cavern, as a sign that I've achieved a high score; you presumably see it as a nuisance, holding you back from swiftly tackling the next cavern!

Yeah, if I understood correctly, this tool was for finding the quickest time for each cavern rather than the highest score (as these aren't necessarily unified in Solar Power Generator). It feels like the trade-off would be the most optimum route that also ended with as little air as possible, so that the countdown time was minimal.

EDIT: didn't notice there was a second page on this thread, so this has pretty much already been covered. Whoops.

Okay, so I revisited the Top Landing using the jump that Danny uses over the green balls in The Bathroom and found some very interesting results:
It appears that if you are doing an Any% (or Warpless) run, which requires losing all your lives bar one before the end of the game then doing the Top Landing item last is ever so slightly faster due to the increased game speed.
However...
If you are doing a Max Lives run then because you lose just one life the game speed is such that it's actually slightly faster to do the Top Landing item first.

Either way the time difference is very minor, so if you have a preferred order of doing it then it doesn't matter *that* much (less than a second difference either way). But speedrunning JSW does come down to applying as many small time saves across the run as you can, so depending on what category you're doing it's an interesting variation to take into account.

Recently I attempted to generate new strats for speedrunners for DOS game Prince of Persia (PoP). The approach was mostly a brute force, yet surprisingly it worked pretty well, although most of the findings were micro-optimizations.
Now I'm trying to do the same for Manic Miner (and if successful, later for other ZX Spectrum games).
Algorithm is pretty dumb breadth-first search:
0. Initially have just 1 state, start-of-the-level.
1. In the beginning of the step, there are N states from the previous step.
2. For every step, for all possible player inputs, generate possible states for the next frame.
3. Deduplicate and score the states (using some heuristic).
4. Keep only N=LIMIT best possible states.
5. Goto 1.
Quite unexpectedly, performance turned out to be a problem. For PoP (actually, I used open source port of it called SDLPoP) my tool could generate whopping 800'000 frames per second, while for Manic Miner it's only 2000, which greatly reduces the scope of what's possible. For MM I guess it's possible to generate a level per night or so, but later JSW plans will need some optimization.
Actually with the z80 emulation library that I initially used it was 200 fps, then I switched to this one and it was much faster. It's pretty hard to find emulator which wouldn't claim that it's "fast" in the description, but it doesn't look like anyone have ever done any comparison.
One optimization that helped a lot (2.5x or so) is not to try to press any keys when Willy is in the air. Turns out that optimization was buggy, so I switched it off for now. Willy can finish jump and start a new one within one game cycle, so checking that in the beginning of the cycle is not enough.
---
Anyway, I've just finished generating the Level 1, here it is attached as .rzx and as .mp4 file.
565 frames total, score is 1724, and I believe that this score cannot be improved.
Will leave level2 calculating overnight.
Peek 2021-03-07 19-00.mp4 lvl1.rzx

Okay, so I revisited the Top Landing using the jump that Danny uses over the green balls in The Bathroom and found some very interesting results:
It appears that if you are doing an Any% (or Warpless) run, which requires losing all your lives bar one before the end of the game then doing the Top Landing item last is ever so slightly faster due to the increased game speed.
However...
If you are doing a Max Lives run then because you lose just one life the game speed is such that it's actually slightly faster to do the Top Landing item first.

Either way the time difference is very minor, so if you have a preferred order of doing it then it doesn't matter *that* much (less than a second difference either way). But speedrunning JSW does come down to applying as many small time saves across the run as you can, so depending on what category you're doing it's an interesting variation to take into account.

Well I initially thought I was done with speedrunning JSW a few months ago when I recorded my last Warpless run, but as I mentioned earlier in the thread we had a new member of the community join recently who spotted a few things we'd missed, and that's helped rejuvenate interest in it as of late. Sometimes you just need new people to stir things up a bit.
The one thing I still never really fully got my head around about speedrunning JSW was how best to optimise deathwarps in the game, as it's a really elaborate balance between the time you save by ditching lives ASAP and the time you save by shortcutting specific rooms. I feel like the route I've got at the moment is a decent speculative one for handling this quandry, but I'm sure a more scientific approach could probably find better parts of the run in which the lives could be ditched.
I don't know if he is aware. I watched Kim Justice's documentary about Smith on YouTube and it seems like he doesn't have a huge amount of fondness for JSW these days, given his rather turbulent history with the game. I believe he's still proud of MM though, and it was briefly floated that we should try to get him involved on the commentary for the MM race at ESA the other month, but it was ultimately concluded that his commentary probably wouldn't be PG enough. 😄

Well I initially thought I was done with speedrunning JSW a few months ago when I recorded my last Warpless run, but as I mentioned earlier in the thread we had a new member of the community join recently who spotted a few things we'd missed, and that's helped rejuvenate interest in it as of late. Sometimes you just need new people to stir things up a bit.
The one thing I still never really fully got my head around about speedrunning JSW was how best to optimise deathwarps in the game, as it's a really elaborate balance between the time you save by ditching lives ASAP and the time you save by shortcutting specific rooms. I feel like the route I've got at the moment is a decent speculative one for handling this quandry, but I'm sure a more scientific approach could probably find better parts of the run in which the lives could be ditched.
I don't know if he is aware. I watched Kim Justice's documentary about Smith on YouTube and it seems like he doesn't have a huge amount of fondness for JSW these days, given his rather turbulent history with the game. I believe he's still proud of MM though, and it was briefly floated that we should try to get him involved on the commentary for the MM race at ESA the other month, but it was ultimately concluded that his commentary probably wouldn't be PG enough. 😄

That's good to know, but I do think the main fun of the mod is working within the new issues that the RNG throws up rather than just trying to patch anything remotely dangerous out.

If nothing else, it leads to moments like this:
https://clips.twitch.tv/DifficultEnjoyableArmadilloANELE--d9OOI0-BOZvobV0
 
I was being slightly tongue-in-cheek. I think we were more speculating over whether the animation of the Chandeliers alternated between starting frames or not.

Well I initially thought I was done with speedrunning JSW a few months ago when I recorded my last Warpless run, but as I mentioned earlier in the thread we had a new member of the community join recently who spotted a few things we'd missed, and that's helped rejuvenate interest in it as of late. Sometimes you just need new people to stir things up a bit.
The one thing I still never really fully got my head around about speedrunning JSW was how best to optimise deathwarps in the game, as it's a really elaborate balance between the time you save by ditching lives ASAP and the time you save by shortcutting specific rooms. I feel like the route I've got at the moment is a decent speculative one for handling this quandry, but I'm sure a more scientific approach could probably find better parts of the run in which the lives could be ditched.
I don't know if he is aware. I watched Kim Justice's documentary about Smith on YouTube and it seems like he doesn't have a huge amount of fondness for JSW these days, given his rather turbulent history with the game. I believe he's still proud of MM though, and it was briefly floated that we should try to get him involved on the commentary for the MM race at ESA the other month, but it was ultimately concluded that his commentary probably wouldn't be PG enough. 😄

Thanks again, everyone, for all of your comments! 👍 To answer RuffledBricks's question - no, I'm not on Discord.
I guess that the time difference has been explained satisfactorily. I did not realise turning the music off made such a difference, plus indeed the initial comparison was of a with-lives walkthrough with a life-lost walkthrough, so there must have been a difference.
I find it amazing how deeply members of the speedrunning community go into detail about how to improve the results, and at the level of perfection RuffledBricks and others must have when playing the game. It's another kind of (wonderful) craziness, a bit like we do here sometimes when we discuss some extremely tiny details of game projects, or pore over the code to find a one-byte saving, and things like that.
The intriguing thing is where the optimisation of walkthroughs will stop. I think that we are now very close to perfect routes and executions both in JSW and MM, possible improvements are minimal, and in many rooms simply not possible any more.
Is Matthew Smith aware of the existence of the speedrunning community and your achievements? I think it should thrill him to know how much time people spend on his two games and how much effort they put into activities related to them - now almost 40 years after he created them...

Has anyone other than @jetsetdanny shared videos of how they got these scores? Curious to know if there's any strats I've been missing from my full game speedruns.

Okay, I've looked into this further and the RTA of Danny's RZX run actually looks pretty realistic.
What you need to bear in mind is the game is 8.6% faster with the music off. I'm not sure if my calculations here are entirely accurate, but if my Any% PB had been played with the music on, it would have been somewhere around 22:15 in RTA - almost a whole two minutes slower.

Also, the difference in speed between running with all lives and running with one life left is 2.9%. So each time a death warp is implemented, it will gradually get towards this percentage until it is reached at Cuckoo's Nest. This happens at around 12:14 in my Any% PB, meaning that effectively 4/10 of the run is being played over 2% faster.
Another thing to note is that my Max Lives speedrunning PB - again with the music off - is 22:19:
https://www.speedrun.com/jet_set_willy#Max_Lives
So this is a much more comparable run to measure Danny's time against, given that it similarly has no deaths apart from Conservatory Roof. It also isn't as optimised as his run.
If I'd had the music on for this run, it appears it would have come in at over 24 minutes. So actually Danny's RTA looks pretty much on point.
 
 
 

Interesting, Spectaculator is one of the accepted emulators for running JSW. I use Fuse (another accepted emulator), so both should be running at the same pace.
I'll ask the Speedtrum Specrunning community (a Discord server dedicated to discussing the speedrunning of ZX Spectrum games) to see if anyone might know where the slower speed is coming from. Or if you're on Discord, I could give you the server link so that you could do so yourself?

Okay, I've looked into this further and the RTA of Danny's RZX run actually looks pretty realistic.
What you need to bear in mind is the game is 8.6% faster with the music off. I'm not sure if my calculations here are entirely accurate, but if my Any% PB had been played with the music on, it would have been somewhere around 22:15 in RTA - almost a whole two minutes slower.

Also, the difference in speed between running with all lives and running with one life left is 2.9%. So each time a death warp is implemented, it will gradually get towards this percentage until it is reached at Cuckoo's Nest. This happens at around 12:14 in my Any% PB, meaning that effectively 4/10 of the run is being played over 2% faster.
Another thing to note is that my Max Lives speedrunning PB - again with the music off - is 22:19:
https://www.speedrun.com/jet_set_willy#Max_Lives
So this is a much more comparable run to measure Danny's time against, given that it similarly has no deaths apart from Conservatory Roof. It also isn't as optimised as his run.
If I'd had the music on for this run, it appears it would have come in at over 24 minutes. So actually Danny's RTA looks pretty much on point.
 
 
 

So I played this on stream for just over an hour and a half today. It was a good lark, although I'm definitely a long way off being able to get a full run in on this version! It's amazing how something as simple as adding RNG to the game can completely re-shape how it plays.
Here's a few initial observations:
Nightmare Room is much harder as I expected. The opportunities to shortcut across the middle of the room are significantly truncated by the randomised cycles. Conservatory Roof is now a boss stage. The rope randomisation is really effective. So much so that I've had to change the order in which I go to Swimming Pool now because it's too risky compared to my normal route. Under The Drive has gone from the 'pixel perfect jump' room to the 'pray that you can even get anywhere near that pixel' room. 😅 We did wonder during the stream whether the Chandeliers' cycles had been randomised too... Thanks again for doing this @Norman Sword, it's ended up being as fun as I expected it to be.
 

So I played this on stream for just over an hour and a half today. It was a good lark, although I'm definitely a long way off being able to get a full run in on this version! It's amazing how something as simple as adding RNG to the game can completely re-shape how it plays.
Here's a few initial observations:
Nightmare Room is much harder as I expected. The opportunities to shortcut across the middle of the room are significantly truncated by the randomised cycles. Conservatory Roof is now a boss stage. The rope randomisation is really effective. So much so that I've had to change the order in which I go to Swimming Pool now because it's too risky compared to my normal route. Under The Drive has gone from the 'pixel perfect jump' room to the 'pray that you can even get anywhere near that pixel' room. 😅 We did wonder during the stream whether the Chandeliers' cycles had been randomised too... Thanks again for doing this @Norman Sword, it's ended up being as fun as I expected it to be.
 

I have thought about trying this. However, there's a few things that one would need to bear in mind.
Firstly, it would be about measuring the speed improvements you would get from losing your six spare lives right near the beginning vs. using deathwarps in the six most beneficial rooms in the game (currently understood to be Chapel, Nightmare Room, Cold Store, Tool Shed, Off Licence and either Cuckoo's Nest or Under The Megatree). Because even though having no lives does make the game run faster overall, it may not be enough to offset shortcutting some of the longer dead-end rooms.
Secondly, figuring out the quickest way to lose those six lives would be significant, as for optimisation it would need to be a death that you could quickly implement six times in a row that's convenient to the speedrun route. Currently I think the yellow monk in First Landing would be best for that if you just walked back in six times from The Chapel, as it's the first enemy in the initial rooms of the game that spawns very near to a door. Yellow Maria in Nightmare Room has a similarly useful position, but that would be slightly off-track from the current route.
In answer to your first question, yes. Because although the score does largely match up with how fast you've played the game in MM, there are of course two caverns where this diverges:

1) The first Kong Beast level, because as you state, going out of your way to dunk the Kong adds a notable portion of time to the completion time.
2) Solar Power Generator, because of the air beam. Technically in this one you'd want to end the level with as little air as possible, so that the run down of the air counter is shorter (yes, I know this is a miniscule time save, but every little counts!).
In terms of life losses, we looked into this briefly for MM but never really figured out a satisfying conclusion to this one, and it might be because you have several fewer lives than in JSW, so the difference is less pronounced. Also, finding a convenient place to lose the spare lives quickly is key and I didn't really manage to find many good ones. The spider at the left of The Menagerie is the first really nearby one, and I found trying to jump into that really awkward.
Beyond that though, there's not really much you can do in MM to optimise it, which is why I find it less interesting than JSW for speedrunning purposes.

Wonderful, thanks for doing this Norman. Will try it out on my Twitch stream later today.

Cheers @jetsetdanny.
So yes, JSW speedrunning is measured by RTA (Real Time Attack) rather than IGT (In-Game Time), given that the latter is quite imprecise - the in-game clock only shows minutes, not seconds - and that many games on SRC (speedrun.com) will take RTA into account even if there is an in-game clock.
Also, there are several variants in JSW gameplay that affect the RTA - switching the music off and losing lives both incrementally speed the game up over the course of the run. And, as we only discovered a few days ago (after my 20:24 run), playing the game on 128K makes it about 1.34% faster than on 48K, so this is now being used in runs given that it should save approx 20 seconds in a completed run (and therefore achieve sub-20 mins if everything else goes right). There's been a few grumbles here and there about this, but in my view it's no different to running Mega Drive-era Sonic The Hedgehog games on NTSC rather than PAL for a speed increase, which is widely accepted in that speedrunning community.
I watched your Max Lives RZX run that you linked and it seems like a good 95%+ of those strats are in my route already, so it looks like it's just a case of execution. But to be honest, just the fact that I've shown that it's possible to reproduce the vast majority of that without rollback means that I'll be pretty happy with whatever my PB (personal best) ends up being, even if there are little movement mistakes here or there.
In terms of running the emulator faster as @IRFsuggested (albeit slightly tongue-in-cheek if I read it correctly), this isn't something that's allowed in speedrun.com rules, and we did have to have a discussion at one point about which Spectrum emulators were permitted, given that not all of them replicate the Spectrum's speed accurately. I'm pretty sure the main reason Danny's run is significantly slower in RTA is because he has the music on. 
So I did record footage of both scenarios and put it together to show a comparison. This can be seen in one of my JSW speedrun tutorial videos here:

https://youtu.be/-shVsGf5-F8?t=1730
The rationale was that because the distance is pretty similar and you lose a little bit of time on the green balls in The Bathroom if you do those first then it comes out slightly quicker (plus you are 7 lives down in the Any% route at this point, so you want to take advantage of the increased RTA speed by doing as much of the mansion in that state as possible). However, your RZX video shows that it is possible (if precise) to do a quicker jump over the balls, so I might need to reconsider. Another reason I liked doing the Top Landing item last was because if you mess up a run early on (e.g. Main Stairway) then it's a quicker reset.
 
This one is definitely a timesave:
https://youtu.be/-shVsGf5-F8?t=1032
The reason being that the warp resets the cycle of the rope, which actually pushes it further ahead by half a second, so you can get up to the top right ever so slightly quicker.
 

A combination of the music being off in RB's video (and on in your recording), with the fact that RB played some of the game with fewer remaining lives, could add up to explain the difference?

The game frame rate and video frame rate have no relation to each other (for games that rely on CPU interrupt frequency there may be differences, but JSW disables interrupts).
The actual game framerate would be somewhere between 9 or 13 fps depending on number of moving objects in a screen, number of lives left, whether the music is on, etc. There's also a difference between ZX Spectrum 48 and later variants as they have slightly different CPU clock speed (3.5MHz for Spectrum 48, almost 3.55MHz for Spectrum 128).
If you record video of the JSW and watch it frame by frame, you'll notice when you go from a frame to the next one, most of the time the image doesn't change. (E.g. if you record at 30fps, and game is ~10fps, only every 3rd frame would show the change).

Cheers @jetsetdanny.
So yes, JSW speedrunning is measured by RTA (Real Time Attack) rather than IGT (In-Game Time), given that the latter is quite imprecise - the in-game clock only shows minutes, not seconds - and that many games on SRC (speedrun.com) will take RTA into account even if there is an in-game clock.
Also, there are several variants in JSW gameplay that affect the RTA - switching the music off and losing lives both incrementally speed the game up over the course of the run. And, as we only discovered a few days ago (after my 20:24 run), playing the game on 128K makes it about 1.34% faster than on 48K, so this is now being used in runs given that it should save approx 20 seconds in a completed run (and therefore achieve sub-20 mins if everything else goes right). There's been a few grumbles here and there about this, but in my view it's no different to running Mega Drive-era Sonic The Hedgehog games on NTSC rather than PAL for a speed increase, which is widely accepted in that speedrunning community.
I watched your Max Lives RZX run that you linked and it seems like a good 95%+ of those strats are in my route already, so it looks like it's just a case of execution. But to be honest, just the fact that I've shown that it's possible to reproduce the vast majority of that without rollback means that I'll be pretty happy with whatever my PB (personal best) ends up being, even if there are little movement mistakes here or there.
In terms of running the emulator faster as @IRFsuggested (albeit slightly tongue-in-cheek if I read it correctly), this isn't something that's allowed in speedrun.com rules, and we did have to have a discussion at one point about which Spectrum emulators were permitted, given that not all of them replicate the Spectrum's speed accurately. I'm pretty sure the main reason Danny's run is significantly slower in RTA is because he has the music on. 
So I did record footage of both scenarios and put it together to show a comparison. This can be seen in one of my JSW speedrun tutorial videos here:

https://youtu.be/-shVsGf5-F8?t=1730
The rationale was that because the distance is pretty similar and you lose a little bit of time on the green balls in The Bathroom if you do those first then it comes out slightly quicker (plus you are 7 lives down in the Any% route at this point, so you want to take advantage of the increased RTA speed by doing as much of the mansion in that state as possible). However, your RZX video shows that it is possible (if precise) to do a quicker jump over the balls, so I might need to reconsider. Another reason I liked doing the Top Landing item last was because if you mess up a run early on (e.g. Main Stairway) then it's a quicker reset.
 
This one is definitely a timesave:
https://youtu.be/-shVsGf5-F8?t=1032
The reason being that the warp resets the cycle of the rope, which actually pushes it further ahead by half a second, so you can get up to the top right ever so slightly quicker.
 

You get seven spare lives in JSW, so you could implement a 'deathwarp' in all seven of those rooms (leaving no dancing Willies on the status bar to slow down the game towards the end).  Though I guess you want to keep one spare in case of 'accidents'!
Jumping up from near the top of the ramp in First Landing into Top Landing is a good way to expire several lives in quick succession (with the advancing Swiss Army Knife), though you have to be careful with the positioning so that you can escape with a vertical jump once you're down to your last life (if you're too close to the Swiss Army Knife when you pop up into Top Landing then you're in an inescapable Infinite Death Scenario!)

I have thought about trying this. However, there's a few things that one would need to bear in mind.
Firstly, it would be about measuring the speed improvements you would get from losing your six spare lives right near the beginning vs. using deathwarps in the six most beneficial rooms in the game (currently understood to be Chapel, Nightmare Room, Cold Store, Tool Shed, Off Licence and either Cuckoo's Nest or Under The Megatree). Because even though having no lives does make the game run faster overall, it may not be enough to offset shortcutting some of the longer dead-end rooms.
Secondly, figuring out the quickest way to lose those six lives would be significant, as for optimisation it would need to be a death that you could quickly implement six times in a row that's convenient to the speedrun route. Currently I think the yellow monk in First Landing would be best for that if you just walked back in six times from The Chapel, as it's the first enemy in the initial rooms of the game that spawns very near to a door. Yellow Maria in Nightmare Room has a similarly useful position, but that would be slightly off-track from the current route.
In answer to your first question, yes. Because although the score does largely match up with how fast you've played the game in MM, there are of course two caverns where this diverges:

1) The first Kong Beast level, because as you state, going out of your way to dunk the Kong adds a notable portion of time to the completion time.
2) Solar Power Generator, because of the air beam. Technically in this one you'd want to end the level with as little air as possible, so that the run down of the air counter is shorter (yes, I know this is a miniscule time save, but every little counts!).
In terms of life losses, we looked into this briefly for MM but never really figured out a satisfying conclusion to this one, and it might be because you have several fewer lives than in JSW, so the difference is less pronounced. Also, finding a convenient place to lose the spare lives quickly is key and I didn't really manage to find many good ones. The spider at the left of The Menagerie is the first really nearby one, and I found trying to jump into that really awkward.
Beyond that though, there's not really much you can do in MM to optimise it, which is why I find it less interesting than JSW for speedrunning purposes.