Dual Programs, Virtual EEPROM

[abarnes]

bump.

Topic: Remotely updating the firmware on our chips (without a serial cable or direct PC connection).

Anyone have some useful input on this?

I'm going to be working on exactly this problem within the next week or two. Our units need the capability to update their firmware remotely, through the use of our cell phone chip. We have 1gb on board flash memory to work with for storing the downloaded firmware prior to burning it. The difficult part, of course, is the unassisted update/bootstrap process itself, and not the download of the firmware to the flash memory.

I'm looking to devise a software-based solution(/hack) which I can burn onto our ZX chips before they are deployed in a remote environment.

I have only marginal experience with the ZX itself but I have a great deal of programming experience and familiarity with assembly, interpreters/virtual machines, buffer-overflow bugs and execution (hopefully i don't have to resort to this), and I've covered the 2 ZX manuals.

So any pointers would be extremely useful. Once I start, my desired timeframe is a working prototype within 2 weeks.

Thanks in advance,
Austin

[abarnes]

p.s. I'm working with the ZX-44a for this project.

[mikep]

I'm going to be working on exactly this problem within the next week or two. Our units need the capability to update their firmware remotely, through the use of our cell phone chip. We have 1gb on board flash memory to work with for storing the downloaded firmware prior to burning it.

What is the hardware connection? Is it serial flash connected to the ZX-44a via a chip select or some other mechanism? What is the process by which the update is initiated?

I'm looking to devise a software-based solution(/hack) which I can burn onto our ZX chips before they are deployed in a remote environment.

I have only marginal experience with the ZX itself

Is someone else providing the ZBasic solution as you have marginal experience? This software update process is non-trivial so is probably not the first thing try out given that you are new to ZBasic. Given your software experience it won't take long to come up to speed.

There would seem to be six parts to a possible solution:

• Ensure that the bootstrap loader is part of the application and is not overwritten by a new version
• Jump from the application into the bootloader
• Read bytes from the source into RAM
• Write bytes from RAM into program memory at the correct address and repeat read until no more data
• Verify consistency and cross-check by using program lengths and/or version numbers
• Jump from the bootloader to the main application code

[dkinzer]

Topic: Remotely updating the firmware on our chips (without a serial cable or direct PC connection).

The ZX devices that use serial EEPROM for Program Memory (e.g. ZX-24, ZX-40, ZX-44, etc.) have the ability to copy a program from a second serial EEPROM to Program Memory at boot time.

You could use this capability if you have radio/cell phone link or some other method to get the new program to the ZX and have it write the new program into the auxillary EEPROM. After confirming that the integrity of the data in the auxillary EEPROM, you call the SetBoot() routine to prepare for loading the new code and then call ResetProcessor() to cause a restart.

Code: Select all

Call SetBoot(bootPinCS, codeSize)
Call ResetProcessor()

The SetBoot() subroutine is not currently documented but it has been in the VM since v1.4.1 (October 2006). The first parameter is a Byte value that represents the pin connected to the chip select of the auxillary EEPROM. The second parameter is an UnsignedInteger value that gives the number of bytes to transfer (beginning at address 0) from the auxillary EEPROM to the primary EEPROM.

The auxillary EEPROM must be have specifications similar to those for Program Memory, e.g. AT25256A or similar.

[dkinzer]

We have 1gb on board flash memory to work with for storing the downloaded firmware prior to burning it. The difficult part, of course, is the unassisted update/bootstrap process itself, and not the download of the firmware to the flash memory.

When I re-read your post, I realized that I had missed the mention of the on-board Flash memory. I assume that you'll want to use that instead of adding a small serial EEPROM as you would need to do for using SetBoot().

Since the code that the ZBasic compiler generates is usually position independent, it may be possible to copy the code for a subroutine to the high end of Program Memory. To execute that code, you'd have to modify a return address on the stack so that it points to a compatible place in the relocated routine. The relocated routine would then have to copy from your on-board flash to Program Memory, being careful to not overwrite any part of the relocated copy routine in the process.

There are all sorts of things that could go wrong with this process and it is emphatically not recommended. If you decide to proceed anyway, you'll have to glean the information needed to do it by examining the listing files generated by the compiler and the usual experimentation.

[mikep]

Since the code that the ZBasic compiler generates is usually position independent, it may be possible to copy the code for a subroutine to the high end of Program Memory. To execute that code, you'd have to modify a return address on the stack so that it points to a compatible place in the relocated routine. The relocated routine would then have to copy from your on-board flash to Program Memory, being careful to not overwrite any part of the relocated copy routine in the process.

I was thinking that the loader can be part of a state machine in main() which is always guaranteed to be at the beginning of Program Memory. That way the loader could overwrite everything else in Program Memory except main function. It may even be possible to overwrite most of main itself as well but that is even more advanced.

There are all sorts of things that could go wrong with this process and it is emphatically not recommended. If you decide to proceed anyway, you'll have to glean the information needed to do it by examining the listing files generated by the compiler and the usual experimentation.

I agree that all sorts of things could go wrong and that is why I cautioned that he really needs a lot more experience in ZBasic before tackling a job like this. It also pays to understand some of the internal workings of the ZBasic virtual machine.

This subject of updating an existing program has now come up 3 times. Once from the original poster, once in my work on RS485 networking where I thought about updating slave nodes from the master (see application note AN-218) and now again in this latest discussion. The SetBoot() function is certainly very useful but it doesn't cope with all scenarios.

[abarnes]

thanks for all of the quick replies!

it looks like i'll have to emphatically ignore your advice (= since this functionality is required, and a board redesign is too costly. manual board modifications are also not desireable. i'm working with an inherited design that didn't necessarily anticipate a custom hardware solution to this problem, so i'm limited to a software solution with the existing hardware design.

i do understand that the VM isn't designed to do this. that really makes it more interesting to me, anyway.

now that i've done the preliminary research on how the VM interacts with the assembly code, i'm quite confident this is achievable.

another interesting thing that can be done with some mangling of the assembly's logic is hot-swapping of the code without restart. while this is tricky, it's certainly manageable. it will be a great proof of concept to start with, though the deployed version would probably depend on a processorreset() for good measure.

for instance,two new functions could be introduced to a given firmware -- a wrapper function and a dummy function (with dummy code that will compile). the wrapper function would call the dummy function whenever wrapper is called.

after a compile, the assembly can be examined to locate the wrapper function and look at the line which calls the dummy function. within the line, the 2 byte offset is given for the dummy function (in the actual bytecode, the high byte and low byte of the calling line are transposed). the exact memory offset of this line is recorded for later use.

using this logic, the code could be engineered such that the compiled bytecode representing a new function (compiled separately) could be typed in by the user over a live serial connection while the firmware is running, and written directly to memory (after bit conversion) at an unused high offset point. this memory would have previously been reserved by using an array declaration of structs containing bounded strings, for instance.

once the user's compiled bytecode is manually typed in and loaded into memory, the previously recorded offset of the line to call the dummy function is overwritten (by running firmware) with a call to the offset of the newly entered precompiled function.

any application calling the wrapper function would now invoke the dynamically loaded function. data could be exchanged through global variables. since the call line is atomic, this hypothetical solution works for multithreaded applications.

there are several obvious considerations here, one being that if variables are introduced by the precompiled new function, this has to be managed carefully.. using either global variables (with the same offsets as the original firmware) or heap-based local variables that the VM will dynamically allocate. also, calls to other functions by the new function would be possible, as long as the new function was compiled within the old code, and then extracted from the resulting zxb. supporting this is kind of overkill though, considering that hotswappable code is already overkill.

some assumptions i've made are:
* the VM doesn't cache the program code beyond one line at a time
* some other things i don't remember after writing this out..

questions:
* is there an easy way to determine the offset of a function/sub from within running firmware? this isn't necessary but it makes for a more elegant final solution
* any major pitfalls other than the expected and necessary awareness that crazy things can happen when one tries to do things such as this?
* i'll think of some more questions, surely. next post.


a more elegant solution for hotswappable code and the desired bootloader uses similar logic but is thread-based.

Thanks again,
- Austin

[abarnes]

ps. with the bootloader located somewhere near the backend of the chip with some buffer room, and the main task powered down, one could overwrite all of main as easily as anything else (hypothesis).

in this way, one can also update the bootloader itself, if necessary, with a second bootloader that's brought in by download.

- Austin

[spamiam]

I love this thread! I have a special interest (academic, not practical, unfortunately) in self-modifying code. Stuff like a program that re-writes itself as conditions change.....

-Tony

[dkinzer]

* is there an easy way to determine the offset of a function/sub from within running firmware? this isn't necessary but it makes for a more elegant final solution

The CodeAddress property yields the address of a subroutine or function. Note that the type of a Program Memory addresses is Long even though it will never be larger than 64K in current ZX devices.

Code: Select all

Dim procAddr as Long
Sub Main()
   procAddr = foo.CodeAddress
End Sub

Sub foo()
End Sub

[mikep]

now that i've done the preliminary research on how the VM interacts with the assembly code, i'm quite confident this is achievable.

It is definitely achievable and there are multiple ways to do it.

another interesting thing that can be done with some mangling of the assembly's logic is hot-swapping of the code without restart. while this is tricky, it's certainly manageable.

The mechanism I'm thinking of does not require mangling of any byte codes or looking up of function offsets. It does rely on the loader being part of main, right at the beginning of the program and can deal with overwriting of the whole program without a reset providing that the first 10 bytes or so are never changed.

It sounds like I should put together a prototype to prove it out.

[dkinzer]

It does rely on the loader being part of main, right at the beginning of the program...

The code for Main() doesn't necessarily start at the same address for all programs. The length of the startup code depends on certain elements of the program. For example, if a program has module-level variables, there is initialization code for those variables in the startup code. All of the module-level variables are zeroed en masse using a single instruction (INIT_VAR) but each String variable is initialized separately using various instructions depending on the string type and whether it is a scalar or an array. You'll also notice that the instructions in the startup code change depending on the amount and types of module-level variables.

Some time ago we had an experimental version of the compiler that put all of the Program Memory data items (whether initialized or not) between the startup code and Main(). This allowed more efficient code to be generated in certain cases but it also introduced a problem for which we had no solution at the time. It is possible that we may go back and resolve that problem in the future.

[mikep]

The code for Main() doesn't necessarily start at the same address for all programs. The length of the startup code depends on certain elements of the program. For example, if a program has module-level variables, there is initialization code for those variables in the startup code. All of the module-level variables are zeroed en masse using a single instruction (INIT_VAR) but each String variable is initialized separately using various instructions depending on the string type and whether it is a scalar or an array. You'll also notice that the instructions in the startup code change depending on the amount and types of module-level variables.

I have now posted a prototype loader that I believe meets the requirements posted on this thread. More work could be done and I welcome comments on that other thread.

The restriction about the variable size initialization code could be fixed by calling a single init function at startup time. That init function is stored someplace after main and performs the necessary VM application initialization functions. Then the number of program memory bytes before main would always be fixed.

[dkinzer]

The restriction about the variable size initialization code could be fixed by calling a single init function at startup time.

I have posted an experimental version (read: not fully tested) of the compiler that implements this change along with a few others. The .zip file contains a description of the changes in the new version.

Mike's description refers to an AppStop function that "reverses an AppStart". It should be noted that most, if not all, tasks should be terminated before beginning an update. Mike's update code does perform a LockTask() but this does not prevent a task from getting control if it is awaiting an interrupt or the expiration of an interval. If all tasks except Main() are terminated, the LockTask() call serves no useful purpose.

[mikep]

Mike's description refers to an AppStop function that "reverses an AppStart". It should be noted that most, if not all, tasks should be terminated before beginning an update. Mike's update code does perform a LockTask() but this does not prevent a task from getting control if it is awaiting an interrupt or the expiration of an interval. If all tasks except Main() are terminated, the LockTask() call serves no useful purpose.

Good point and exactly correct although I do not have an example of this in my test applications. I will amend the code and remove the task locking from the loader and try out the new compiler. It is definitely a cooperative relationship between the target application and the loader.