Presentations

Quick hacks!

After completing a block of work like the presentations, there are often a few things to clean up, but most of the things that needed to be done already have been. So this is usually a time that I flail about uncontrollably, wondering what to do next. My best solution to this is to just do something! A quick hack to fill a gap in the APIs or creating something that's not great but really is just good enough, fills that space. Not only does it give some direction, but it also produces something that is usually useful.

CompressJPEG

After the 2021 presentation, I had finished all I needed, so I looked at some of the things that were missing. I had spent a lot of time on the graphics system and because of that, I had a reasonable understanding of how easy it should be to create a CompressJPEG module.

The CompressJPEG module also is similar to the Squash module in that it takes some data as input, and produces some output. And, like the Squash module, the Python interface I was using does not support streaming, so we need to buffer the image in preparation for outputting the content at the end.

The implementation of the CompressJPEG module is nearly complete. The standard RISC OS 3.5 SWIs are implemented, and the CompressJPEG_WriteLineExtended from RISC OS Select, but there's no CompressJPEG_Transcode. That could probably come later, but it won't be a commonly used SWI call so it's better to leave until it's needed.

WebColours

WebColours is a little bit unique in that it's largely there for me. I had used the original WebColours module for my presentation software in the earlier presentations, but it was small and easy to implement, so I threw it into the RISC OS Pyromaniac system. It's also relatively easy to test as it has a very basic interface of colour numbers or names going in and the opposite coming out.

Speak

For a while, I had wanted to create an interface for the Superior Software Speech! module. However, in my own archives and online, I couldn't find it documented or even a copy of it. On macOS, it isn't too hard to control the host text-to-speech system, so it would have been really handy to provide an interface for Speech!. But as I had not found any information about it, I decided to try implementing Jonathan Duddington's Speak module instead.

The Speak module's interface is a lot more advanced than the Speech! module. It actually has a reasonable usage on Linux these days as eSpeak. The SWI interface isn't too hard to provide, so I set about creating a stub module that could just take the basic calls and get the machine to speak. Although it wasn't hard, there were a lot of SWI calls that weren't well documented. I dug into some of the source code and pulled out some of the bits that seemed right.

As a result, there is now a relatively complete implementation of the Speak module in RISC OS Pyromaniac. It even handles the different voices provided by macOS. In theory, an implementation could be produced for Windows, but I've not spent the time on that. It wasn't actually that 'quick' a hack in the end, but the results are quite fun. It also makes RISC OS Pyromaniac more accessible, which is a significant feature.

SpellMod

Having worked on speech, another module that I thought might be a nice quick thing to put together was the SpellMod spelling checker, as used in Impression. This module had its roots in the BBC InterWord series, I believe, and in some places its age really did show.

Steve Drain had done an excellent job of decoding a lot of the SWI interface and produced a StrongHelp manual that was invaluable in creating a module. He sent me a collection of his findings in the middle of last year to help with the process.

The SpellMod, like Speak, uses implementations in RISC OS Pyromaniac to allow different ways of working. There are only two implementations - a 'null' implementation and a 'wordsfile' implementation. The 'wordsfile' implementation allows it to use a file that contains valid words to check against - /usr/share/dict/words. It can also apply different algorithms for the anagram and Soundex matching of the words.

The SpellMod experiment is still sitting on a branch because it needs more work. I'm not happy with the dictionary saving code, in particular. I am pretty sure it will produce dictionaries that are valid according to the documentation, but which would not be usable by the Classic SpellMod. Plus there are no automated tests yet, so it isn't reliable enough to get merged. I shall return to it at some point, but it isn't a core module required by RISC OS.

Euclid

I've been interested in getting Euclid to work on RISC OS Pyromaniac for a while. Euclid was a 3D graphics editing application from RISC OS 2 days. It was quick clever, so it's a daunting process, but I did start it... and that's about it. The experimental branch has a Euclid module that has most of the SWIs documented and returning an error. When I come back to it in the future, it will be easier to continue. That's how some experiments go... they get a little way and then fizzle out because there isn't a lot depending on it and something more fun has come along.

Joystick

I had wanted to extend the Joystick system for a while. My plan had been that where Select 4 was starting to introduce an improved input system, Select 5 or 6 would probably improve the Joystick support. Joysticks - basically gaming device inputs, but such inputs are also vital to people with accessibility needs - had not been touched since the A3020 days, and realistically that was just a formalisation of what other third parties had been doing because the ADC interface for the BBC was no longer suitable. The interface as documented by Acorn is rudimentary and hasn't really changed since then.

I wanted to extend this and I made a few suggestions for revamping the interface on the ROOL forums which were met with a distinct lack of enthusiasm. I decided to just do something and see what came of it. "Not much" is the answer. I've got a small module and it doesn't do much.

One of the problems with creating a module is that the input for it is a bit fiddly to test manually. MacOS doesn't offer a direct interface for Joysticks, which means that all the work of reading the joystick's state must be performed by the program. I had an idea that if I could get an analogue input instead, that would actually do for the primary stick input. This gave me an idea for later in the year when I was working with some real hardware interfaces.

In any case, the Joystick work is parked in its experimental branch until I have time to come back to it. It's not a module that you'll usually need to test things against, but there is at least a partial implementation for when I need it.

Graphics as PDF

Sometimes the "let's see if this works" is so successful the result is a simple "Oh, yes it does". One such example was the experiment to see whether the Cairo graphics system could use a PDF as its output. I had already demonstrated that using an SVG was possible and even useful for the output, and PDF seemed like it ought to work but "surely it can't be that easy?".

Yes, it is that easy, and it literally just worked the first time. 49 lines of code - including the lines that made up the test - and we can now create PDFs out of the graphics display.

Is it useful? That's a different question, but it is rather neat.

The PDF file linked shows the desktop as a PDF. One thing that probably won't be noticed, though, is that the entire sequence of operations that made the desktop up is present. This includes the startup banner, which is immediately hidden by the rest of the desktop. It might be useful to hide completely covered content, but it is just as likely it could be a useful feature to someone.

Block devices

At the start of the year, I thought it would be good to start on one of the longer term projects since I had a good amount of time to work on it. I've been wanting to revamp the block device interface for some time. But to do that, I need to have a baseline interface that I can work to - the existing interfaces need to be able to work so that I can show how to transition away.

The CDFSDriver module had been created back in October 2021, together with a partial implementation of a CDFSSoftPyromaniac. It had its own way of accessing block data in the host system, and I wanted to make that more generic. The internal code used for accessing file data was moved out of the module and made more generic so that it could be used by a new ADFS module.

The ADFS module provides block device access using the FileCore registered interfaces, allowing it to talk to the host block devices in a very similar manner as the CDFSSoftPyromaniac driver does internally. The whole process of registering the module and making it work is one which resulted in much bashing of my head against the table because the system is so awkward - hence why I wanted to be able to replace it.

However, the result isn't so bad when it works. The Pyromaniac ADFS module can register with FileCore and allows disc images to be mounted. In the process, I discovered that raw disc images read directly from hard discs and disc images created by RPCEmu are not necessarily compatible. There is a header at the start of some RPCEmu images which means that they are not quite the same as raw images. There is special logic in the new ADFS module's disc mounting to detect and allow for the RPCEmu formatted images.

Having got some working block devices that can be used with the existing system, it would now be possible to start creating a replacement interface - which I didn't do because I got distracted by...

Jan Vibe's graphical ditties!

Someone - possibly Dave Thomas - pointed me towards a GitHub repository which contained all the graphical programs from Jan Vibe. These were little BASIC programs that created some very pretty images and sometimes animations. Jan's graphics were a staple of the Acorn User programming pages (and later cover discs) and never disappointed.

Because they used the graphics system in interesting ways, and because they were themselves pretty interesting, I had wanted to use them for testing parts of the VDU and graphics system. So I tried these out, and we got a lot of failures. I had some simple shell scripts which I ran on my MacBook which went through all the files and created videos of them running, with a limit of about 2 minutes. Some of the programs ran forever, so I had to limit them.

There were quite a lot of fixes to the graphics system to make sure that we were really doing the right thing. There are still a lot of his programs that don't work right. That is not surprising as some features remain that are not supported, like redirection to a sprite. But many mistakes were corrected and in most cases, new tests were created to ensure that those failures don't regress in the future.

Not only were the results really pretty - there are over 500 videos linked from my GitHub repository for most of the programs - it fixed a lot of problems that I knew must be there but had not had the time to exercise yet. The VDU system is very complicated and there are subtleties (and some not so subtle things) that are often missed.

Although there are lots of pretty ones, the Diamond Tree really made me smile, and was one of the nicer animations as well.

Download video

Better still, the graphics system in RISC OS Pyromaniac is entirely based on the Cairo graphics library, which means that changing from generating a bitmap to generating a vector graphic is just a matter of changing the output format. The same Diamond Tree can be just as easily rendered as an SVG.

At the same time, I began to see how easy it would be to integrate running programs with the host desktop. Not in any particularly clever way, but just to see whether it was possible and practical. On macOS, there are 'Quick Actions' which are associations for files that you can place on the explorer menus. Associating a command which runs the application and then using the program from a menu makes it easy to launch the programs.

One example of running one of his programs using the 'Quick Action' also shows that the mouse works exactly as you expect.

Download video

Classic APIs

There are several modules that are used in RISC OS which are a little more specialised, but if you need to use them, it's good that they are there. My policy in developing RISC OS Pyromaniac is to be aware that features are missing and to implement them when they are needed, or when it seems fun. And if there is some part of the system that's difficult, just leave a note in the code, or a warning and error, so that it's clear that it's not working yet.

Filling in these APIs is important for me to feel that it's a more complete version of RISC OS, and useful (obviously) if you need to test with them.

ShareFS, Freeway

There are a few modules for the Internet stack which are very useful to have available, even if they aren't full implementations. ShareFS and Freeway are two of these modules. In RISC OS Pyromaniac it's important to have an interface that allows you to use the system as if it were working. That way you can find the problems in your code. But beyond responding as you expect, it doesn't need to work completely right. Eventually, these modules might be expanded so that they do everything you need, but it's not necessary for much of the testing.

The ShareFS and Freeway modules are good examples of this principle. They merely track the state of the resources that you register with it and return just like the real modules. But they don't communicate with anything.

This means that you can 'share' a disc and it appears in Freeway's list. You cannot mount it and the information goes nowhere as it's all internal, but that's good enough to prove that the interfaces are working.

charles@phonewave ~/projects/RO/pyromaniac (master)> ./pyro.py --load-internal-modules --command gos
Supervisor

*share $ MyDisc
*fwshow
Freeway types/objects (* = local):

Type 1 (Disc):
  127.0.0.1        *MyDisc

Of course, the interfaces are all tested as well, so I'm pretty sure that it works as I need it to.

InetServices

The InetServices module is used by the RISC OS Select Internet module to provide its lookups for error messages. I had been testing out the experimental DCI 4 driver (which still isn't working well enough to merge in yet), and so it needed to be implemented to get the same behaviour.

Whilst InetService module is largely used for error lookups and service decoding, it also has a feature where it can produce a menu of the available services. I created it largely for my network clients and daemons back in the late 1990s, but to be complete it should be able to produce those menus.

Fortunately, when I created the FontManager_MakeMenus calls, I added did so with a class of BufferedMenu because I expected it would be a thing that I would need in the future. The same mechanism for creating menus is used in InetServices, which made the implementation a lot easier than it would have otherwise been.

ResourceFS

ResourceFS isn't generally a module that you would expect to need for applications. Most of the time you refer to files on ResourceFS with Resources:$.Resources..., and that's all that you care about. In RISC OS Pyromaniac that's easily emulated by just creating a Resources$Path variable to point to a directory that holds those resources. However, if you have a module that wishes to store files in ResourceFS, it will fail without the module.

Like the ShareFS and Freeway module, the implementation of ResourceFS is limited to just tracking the state of the registered resources. This means that you cannot actually access them through the filesystem, but all the SWI calls work as expected. Whilst testing some RISC OS Classic modules it was found that the modules would not start without being able to register with ResourceFS. That's not very nice and whilst I had tried to make the modules resilient to this, I clearly missed some cases.

Similarly, the Internet 7 module which I had been testing needed to be able to register with ResourceFS. The simplest way to do this was to create a module that took the list of registrations, stored it away, and didn't error.

That's pretty much all that the ResourceFS module does. It does have commands to save the registered files, which is useful if you need to extract the resources from a module.

TimerManager

In RISC OS Select much of the work was to make the system more maintainable and future-proof. Part of this was reducing the size and complexity of the Kernel and making the system more modular. The TimerManager module was part of this work. Removing the timer from the Kernel and into a module was intended to make the system easier to retarget to different hardware. Instead of worrying about the complexities of the Kernel, a relocatable module could be produced, just like any other module. Together with this, the TimerManager incorporated the ability to handle more than the single timer that the Kernel required, and with greater flexibility for its timings.

Under RISC OS Pyromaniac the 'timer' is largely driven off the host system clock, but it does not have to be this way. The TimerManager allows the same interface as the abstracted Select Kernel used, and passes through the regular OS_ClaimDevicevector interface. Clients can register for timer events and they will be dispatched appropriately.

The old RISC OS Select documentation was dusted off and converted to the PRM-in-XML format so that it can be included in the Pyromaniac documenation. The module is also tested to make sure that it's doing the right things.

As RISC OS Pyromaniac is largely focused on higher level testing, the TimerManager won't be used that often. However, its addition does begin to bring in more of the device vector system, to allow mocked calls for testing components that need the interface. Of course, many components wouldn't need that as - being a hardware interface - it's not useful on a software only system.

UtilityModule

There are always things that you liked about the system which got lost to the midst of time. Within RISC OS Pyromaniac, there are a few interfaces that are configurable so that you can return to those times. The sound system was one of those cases, largely because I implemented it based on what I remembered - and what I remembered was the BBC scheme of scheduling notes. So the sound system can work in the way it did on the BBC, or as it does in RISC OS.

In the last year, I had been reminded that 'Deleting the UtilityModule is foolish'. This is an error which I don't think you can cause in modern RISC OS for a few reasons, and in RISC OS Pyromaniac it wasn't a problem - if you told the UtilityModule to die, it would. Clearly, this is a terrible thing and I had to restore the old error message.

It's now an option. Not an especially useful option, but an option nonetheless:

OwnerBanner

Whenever I run the system in demos and want to show that it is running RISC OS Pyromaniac, I had to both enable issuing the system banner, and load the OwnerBanner module which provides it. This is a bit tedious and as a final quick hack in the year, I put together a simple implementation as a PyModule. So where previously I had to load the old module from RISC OS Select:

charles@phonewave ~/projects/RO/pyromaniac (master)> ./pyro.py --load-module modules/Banner,ffa --config kernel.reset_banner=true
RISC OS 7.43 (08 Jan 2023) 3MB

RISC OS Pyromaniac

I can now just use the internal version:

charles@phonewave ~/projects/RO/pyromaniac (master)> ./pyro.py --load-internal-modules --config kernel.reset_banner=true
RISC OS 7.43 (08 Jan 2023) 3MB

RISC OS Pyromaniac

Which means one fewer external dependency, which is nice. At the same time, I converted the old RISC OS Select documentation to PRM-in-XML so that it was properly documented.

Unicorn 2

Underneath Pyromaniac lies the Unicorn engine. This is a derivative of the QEmu emulator and often lags behind modern developments. Unicorn 2 upgraded the QEmu implementation from a very old snapshot of QEmu to a much newer version. Because Unicorn is highly customised to allow it to be used as a general purpose emulation system, rather than a system emulation system, this introduced a lot of problems.

The work integrating Unicorn 2 into RISC OS Pyromaniac was not large, as the change is mostly a drop-in, but after it was done there were many execution problems. Privileged modes, and mode transitions, in Unicorn 2 under 32bit ARM were not working properly at all.

There was a lot of back and forth with example programs and tests, and ideas about where the library had problems. Eventually though, having had lots of support from the Unicorn developers, RISC OS Pyromaniac ran successfully with Unicorn 2. This also allows the configuration of different CPU models in RISC OS Pyromaniac, as these are now exposed by the API.

Whilst in development Unicorn developers were supporting Python 2, so although Pyromaniac was running with Unicorn 2 at that time, the release version of Unicorn 2 cannot be used. Pyromaniac will need upgrading to Python 3, which is a not insubstantial amount of work.

There's a small video of RISC OS Pyromaniac running under Unicorn 2, all the way to the desktop and out.

Download video

For now, Pyromaniac runs with Unicorn 1, but is future proof.

Git

Back in early 2020, I created the first version of the *PyromaniacHostCommand, which allows the running of commands from the host system from within Pyromaniac. The output from the command you ran would be processed by Pyromaniac, and sent to the RISC OS VDU system, just like you would expect from any native command. The ANSI sequences can be processed and are turned into ColourTrans or VDU commands so that the command even has cursor control and colouring as you might expect of a RISC OS command. When executed, the working directory would be set to the current RISC OS directory in the host filing system, so the operations you performed would write files to the expected place too.

This was sufficient to create a 'fake' Git client invocation, using some aliases to make running git on RISC OS Pyromaniac relatively easy. Of course, you were directly operating on the host git tool, so filenames had to be correct for the host system and you had to take care of translations of the alphabet - but it worked. I created a very fake example as a video, to show how this might be used. It was interesting, but not that useful. You couldn't commit things because that would mean running editors and having input processed (which was a much bigger problem). The ANSI code parser was simple enough to handle colouring but there are a lot of operations that it just didn't understand and a full console editor was not possible at the time.

But having shown that the operations could work, I decided to write a module that did all the translation work for you, to look like a real RISC OS command. The PyromaniacGit module became an interface to the host git and handled configurations and translations for you. It even interfaced with the host editor so that we could launch nano to edit files. It did this by handling the editing itself and then passing the file that you edited to the git commit message.

And then I moved on to other things, and other than keeping the git-experiment branch up to date with the latest master now and then, it didn't get any more work - until late in 2022.

There was a meeting which I attended which discussed how to give RISC OS users an understanding of Git. It was interesting, and I dusted off the branch to see how far I had got. It was surprisingly functional, although the limitations I'd known about before were still there. Many of the commands that had been implemented worked fine, but many were limited and some had become broken by changes in the host Git version.

The original 2020 version of PyromaniacGit only supported the most common commands, and even then it might not support all the options. In some cases, the commands supported no options at all and just had an options list that said "FIXME". The module that I'd designed was actually pretty well thought out and I was very grateful to past-me that it was relatively easy to add new commands. Many of the environment variables had been defined, to allow them to be mapped through from RISC OS system variables, and some of them had proper translations for the RISC OS filenames.

The intention with the PyromaniacGit module is that it provides an interface that looks as close as possible to how a RISC OS git works. That means...

• It should look similar enough to regular git that online instructions work as you expect.
• It should be able to store files with RISC OS types using the NFS format.
• When you use filenames, you use them in RISC OS format, and it translates them to the host system as necessary.
• When you use labels like branches, tags or repository names, they should be translated from the current alphabet to UTF-8.
• Messages supplied to git (for commits and the like) should be given in the current alphabet and translated to UTF-8 when passed to the host git.
• Output from the git client should be translated from UTF-8 to the current alphabet.
• Output from the git client should hide or understand the ANSI sequences to make it look similar to the host git.
• Configuration through the environment variables should be translated from the current alphabet where possible.
• Configuration through the environment variables using RISC OS filenames should be translated, and where multiple directories are used, comma separation is used instead of colons.
• The password prompts should be honoured properly.
• Authentication should be able to be stored in a credentials file (and disablable for use on public systems).
• Editing messages should be possible (cannot use ee for external edits, but we can have similar editing).

There are some things that are desirable for the RISC OS git client that are not currently possible in the PyromaniagGit module...

• The output should use the RISC OS filename format.
• Handling of rooted filenames should be supported (ie the :<filename> syntax), as this conflicts with RISC OS filename conventions.
• Configuration through the .gitconfig file should be possible somehow (it's too likely for someone to escape the RISC OS system like that - and this is still possible in the repository configuration files).

And that's aside from the commands and switches that are not currently supported.

The main things that aren't supported at the minute are many of the back-end plumbing and advanced commands, and commands that relate to submodules and configuration.

So what can we do?

• Create repositories.
• Clone repositories from remotes.
• Push and Pull changes (with and without authentication).
• Create and manage branches and tags.
• Commit code (with message editing).
• History operations like logs, diffs, and annotations.
• Merging operations like merge, rebase and cherry-pick.
• File operations like add, rm and mv.

That's not everything that it can do - there are many more things that are more specialised - but you get the idea.

The process of running the Git commands is very similar to that of the regular host commands, except that we also translate a number of the environment variables and add in some configuration parameters to make the results appropriate. The command arguments are translated from RISC OS format to host format, and the command is run on the host with the output streamed through the ANSI parser. The RISC OS VDU codes and calls to ColourTrans are used to make it output as expected, and the text has the current alphabet applied to it. If the user presses escape, the process is sent an interrupt. The general processing looks something like this:

When authentication is needed, there are a few more hoops to jump through. We want to be able to perform the authentication within RISC OS's domain, rather than within the host domain, so we have a special tool which handles the request for the user name and password. It connects to a server that Pyromaniac runs, and sends the request from Git to the configured UserInput implementation. By default, this is just the VDU stream and OS_ReadLine but it can be configured to prompt in a dialogue box when used in the UI. The process is a little more involved but quite manageable and reliable. The steps that we go through are thus:

A similar server is used to provide the editing functions too. The tool requests that we exit a file, instead of getting input from the user. When the request to edit the file is received, the TextEditor implementation is used, so we get the same interface that the *Edit command uses or the text viewer in the explorer.

Git is hugely complicated, so it's not going to be possible to implement all the features, let alone describe how they work. But there is some documentation on the Pyromaniac website which describes the PyromaniacGit module and the system variable mapping that has been used.

From the original meeting that sparked returning to the PyromaniacGit work, I had agreed to produce a video of using Git with macOS. The video was relatively easy to do, and I created a second video that shows using git on RISC OS Pyromaniac, using shell.riscos.online.

Administrative

There were a few things that I did which weren't directly related to the Pyromaniac tool itself.

New documentation

The documentation for RISC OS Pyromaniac isn't great, largely because there aren't any real users, but I am trying to treat it as a real product, with releases, change logs and documentation that goes with it. So I've made more efforts to try to document some of the interfaces and parts of the system so that anyone using it has the information they need to work with it.

The main Pyromaniac site has always included documentation about the system's features, the changelogs, configuration options and the tracing options available. These have been updated occasionally throughout the year as I've done new deployments of the cloud systems.

I spent a little time making the API documentation, which was previously only available in a zip archive, available on the main site as well. The new PRM-style documentation is a conversion of the documents generated by the build, but integrated into the main Pyromaniac site. This means that the new documentation that has been created is now easily accessible and can be directly referenced.

Twitter

In March I decided to try out using Twitter to see if it was a useful way to show off the things I was doing. Since early in the RISC OS Pyromaniac development I've been keeping screenshots, videos and other captures of the progress. This is a habit I picked up during RISC OS Select development as sometimes the snippets of programs, images and others became useful to send to people as either demonstrations, publicity or just to give examples of things that might be useful. I found it invaluable when producing the earlier presentations, to have lots of images to work from, and would regularly post them to friends to show the silly things that I had been working on - just like I would have done in the past on IRC.

Posting the images out on Twitter - just as a way to show off some things - became another nice checkpointing exercise. If I created something that I thought was cool which worked, I could just post it out. Doesn't matter whether people pay much attention - what I'm doing is a little niche - but it's out there. Much of what I've written about here has been seen on Twitter already.

Changelog areas

As there are more moving parts in the Pyromaniac system now, the changelogs have become a little unwieldy. It's hard to tell whether a particular change is going to affect the use of the tool. In my day job, some of the tools we deploy are amalgams of many different components, and their change logs describe them in terms of the areas that are affected. This makes it easier to tell whether the change matters or not to the end-user.

To make the logs in Pyromaniac a little more manageable, they were restructured to use this scheme as well. It means a little more work during the release process to determine which areas have been updated. However, as the release already involved tidying up and merging entries anyhow, this wasn't so onerous.

The result is a change log which includes the area that was affected in each line. There are a set of general areas that are used for many of the core operations, and the name of the module is used for the PyModule interfaces. I try to make each section describe the areas in ascending order - from the lowest level components at the start of the list, to the highest level at the end.

The point of recording the changelog is to make it clear to a user what changes might affect them, so this sort of change makes that process a little easier.