ZBasic Application Self-Update

[mikep]

This example code is distributed under a BSD license and can be used in your application providing you maintain the attribution to Oak Micros. The code is provided ASIS without any kind of warranty. The next append describes the self-updating process and loader code that performs this function. Two example programs are provided.

[mikep]

This text is extracted from comments in the loader code. This code should not be used for production purposes. It is provided as an example only.

Requirement
The requirement is to provide a software mechanism for loading a new ZBasic application from an existing application in order to update code. This requirement usually occurs because the ZBasic application is executing remotely and it has to make the decision when to update as user cannot. Some examples include:

• RS485 Slaves - only the master can decide when and how to update slaves
• Remote cell phones

Loader Description
There are two main parts to the ZBasic update mechanism; a loader that is actually invoked via Main() and the target application.

The target application should expose 3 public subroutines. Each takes no parameters and are as follows:

• AppStart - executes application startup policy that should only be executed once such as starting threads and creating resources
• AppStop - reverses an AppStart and frees resources
• AppMain - main line of application - usually a loop that doesn't terminate unless an application update is required

The update mechanism provides a method to decide when an update should be performed using the IsUpdateRequired() function. For the purposes of this example code, the user typing a U on the console is sufficient to start the update. For production code a more sophisticated mechanism might be used such as a special message from a "master". The target application can call the IsUpdateRequired() function to determine if it should exit the AppMain subroutine and return control to the loader.

The loader has 3 states (appRunning, updateApp, and updateLoader) which determine whether it should execute the target application or load an update. AppStart, AppMain, and AppStop are called from the appropriate states as needed.

In order for the loader to update the target application, it needs to retrieve the new code. For the purposes of this example, the code is stored in Program Memory as part of the loader and target application. More sophisicated implementations will load this data from flash memory, SPI or I2C connected EEPROM/FRAM or from some kind of network. Any data sent via a network should have a CRC that is verified by the loader before the data is used. The update code is stored using a ByteVectorData and is compiled into the final application using the "Source" filename mechanism for initializing Program Memory. A separate program named "zxbdump.exe" can be used to read a .ZXB file and dump the ZVM instructions to stdout in a suitable format.

Two example programs (App1 and App2) are provided with this loader. App1 is similar to the "Hello World" sample this is supplied with new ZX devices and alternately flashes the "red" and "green" LEDs while writing to the console a hello world message. App2 is a slight variation that uses PWM to control the brightness of the "green" LED and flashes the "red" LED each
time the "green" almost goes out. These examples should build and execute on any ZBasic target platform but were tested on the Oak Micros ZX-24ae.

Project files are provided with each example. A listing file is produced with each compilation which can be used to check the generated code and the size of main(). The constant FIRST_PAGE_SIZE must be larger than this.

The build.bat command file can be used to build these two ZBasic programs, execute zxbdump and correctly add the ZX VM instructions into Program Memory for each application. By using build.bat to build App1 and App2, App1 can be used to update itself to App2 and App2 can update itself to App1. The basic procedure is to load, for example, App1 into a device. Then press U on the console and it will update itself to App2. If you press U again, then nothing more happens and an error message is displayed. Here is some sample output:

Code: Select all

  ZX24ae says...
      Hello, world
  UUStarting application update.
  Finishing application update.
  ZX24ae says Hello, world
  ZX24ae says Hello, world
  UUNo application update available.
  ZX24ae says Hello, world

Functional enhancements in the loader can be deployed by first updating the the target application with the new loader and the same application code. The application can then be updated using the new loader implementation.

The loader has a RAM overhead of 12 bytes and uses 384 bytes of heap. The RAM could be reduced still further to 2 bytes at the expense of larger and less maintainable code.

Restrictions and Limitations
This example update mechanism does not:

• Verify the target CPU against the one required by the update code
• Update the contents of EEPROM or RAM
• Update the ZVM firmware
• Show how to check for target application versioning
• Provide a robust way to cope with a different size for the ZXB initialization code - i.e. main should be at the same address. A change to the way ZXB programs are built may help here.
  Edit: see http://www.zbasic.net/forum/viewtopic.php?p=4375#4375 for a resolution to this last restriction.

[abarnes]

Thanks Mike, this will be a great reference when we implement the remote update function of our platform.

For further reference on this topic: Forum post discussing possible approaches

I suppose we can just continue the discussion here (if any), rather than the former thread, now that we have some proof-of-concept code to center the discussion.

[mikep]

I suppose we can just continue the discussion here (if any), rather than the former thread, now that we have some proof-of-concept code to center the discussion.

I would suggest we discuss specific comments on this approach in this thread. Discussions on other approaches should continue on the previous thread or a new thread.

[dkinzer]

One of the advantages to this approach is that it work on all ZX devices irrespective of whether Program Memory is in external serial EEPROM or in internal Flash. In contrast, the SetBoot() method only works with ZX devices that have Program Memory in external serial EEPROM.

With regard to the devices that have Program Memory in internal Flash, it would be more efficient to perform the PutProgMem() calls on Flash page boundaries with a byte count that is an integral multiple of the Flash page size in order to avoid read-modify-write operations. For all currently available ZX devices that use Flash for Program Memory, the Flash page size is 256 bytes.

[abarnes]

One of the advantages to this approach is that it work on all ZX devices irrespective of whether Program Memory is in external serial EEPROM or in internal Flash. In contrast, the SetBoot() method only works with ZX devices that have Program Memory in external serial EEPROM.

Precisely the goal ++ other external sources such as external flash nvram.

We're working with a very capable platform for this to have been implemented so trivially.

[dkinzer]

[...]for this to have been implemented so trivially.

Mike just makes it look easy. That's the hallmark of a professional.

[mikep]

With regard to the devices that have Program Memory in internal Flash, it would be more efficient to perform the PutProgMem() calls on Flash page boundaries with a byte count that is an integral multiple of the Flash page size in order to avoid read-modify-write operations. For all currently available ZX devices that use Flash for Program Memory, the Flash page size is 256 bytes.

There is more work to do with sizes of the two buffers used in the loader. One buffer is used to overwrite the loader as the last thing that happens (all in one operation) and the other buffer is used for the remainder of the program that is not part of the main loader function.

One of the missing checks is to verify if the size of the new program is less than the size of the first buffer. This check was removed from the code because the optimizer would eliminate it with the current implementation of reading the new program from program memory.

[mikep]

This example code is distributed under a BSD license and can be used in your application providing you maintain the attribution to Oak Micros. The code is provided ASIS without any kind of warranty. The next append describes the self-updating process and loader code that performs this function. Two example programs are provided.

I have made some small updates to the code based on Don's input with a few other minor changes

[Don_Kirby]

Impressive work Mike. Thanks for sharing it.

Perhaps this should be an App Note.

-Don

[abarnes]

Perhaps this should be an App Note.

I second that.

[...]for this to have been implemented so trivially.

Mike just makes it look easy. That's the hallmark of a professional.

I hope this wasn't misunderstood. I'm very new to the platform and I'm impressed that we could discuss this and three days later one of the two most active developers has already produced a solution, and even then as more of an afterthought than an undertaking. Of course, this was originally discussed amongst you guys a year ago, so maybe that isn't exactly three days, but a year ago a hardware solution was proposed and support was immediately implemented as well, putting this issue to rest at the time.

To clarify, I was referring specifically to the capability of the platform for such an elegant solution to be formed. Some VM platforms introduce complexities that would make an equally functional bootloader much less trivial to implement. The fact that platforms tend to introduce complexities that can make this type of functionality less trivial to implement is partly evidenced by Don's follow up with the tweak to the VM to enable the proposed solution to retain its elegance.

Usually making a firmware capable of remote self-update involves a host of other issues. The very fact that the platform is VM based makes self-updating of the application code more plausible. For instance: it's an order of magnitude more difficult to remotely update the VM itself, though doable. This would be particularly difficult for an outside developer, given that I/he/she don't have the proprietary code.

BTW, that would be a nice feature but it definitely isn't in our project goals at this stage. When we deploy these units we'll be trusting the VM to be relatively bug free, and hopefully not "need" new functionality that only a VM update could provide.


Another question:
(1) How can the firsttime variable be set so that it appears that the firmware has been updated through traditional means? In other words, what's the memory location of the underlying variable, and what are its possible values? Within the bytecode it appears that this is a proprietary VM call, which shields the memory address information through indirection.

I suppose one option would be to poll all of the memory before and after calling firsttime() from within running firmware and doing a diff, but that sounds rather tedious.

(2) And are there any other little issues like the firsttime() thing to be aware of when "abusing" the VM this way? Are there other variables in persistent memory that might be relevant?

- Austin

[mikep]

I hope this wasn't misunderstood. I'm very new to the platform and I'm impressed that we could discuss this and three days later one of the two most active developers has already produced a solution, and even then as more of an afterthought than an undertaking.

I wouldn't call 8 hours work over 3 days an afterthought. I understand your meaning and don't take offense. This idea was something that I had wanted to implement ever since I wrote the RS-485 application note (AN-018) that first mentioned self-updating of an application.

BTW, that would be a nice feature but it definitely isn't in our project goals at this stage. When we deploy these units we'll be trusting the VM to be relatively bug free, and hopefully not "need" new functionality that only a VM update could provide.

In my experience the VM is rock solid. Most of the changes are enhancements and the number of defects found is this amount code has been very small compared to standard industry practices. It is widely recognized in this community that Don has done an outstanding job with the implementation of ZBasic.

(1) How can the firsttime variable be set so that it appears that the firmware has been updated through traditional means?

This data is held in the first 32 of the device EEPROM. It would be easy to set using the PutPersistent routine.

(2) And are there any other little issues like the firsttime() thing to be aware of when "abusing" the VM this way? Are there other variables in persistent memory that might be relevant?

As noted in my restrictions, I do not set the device EEPROM. This will need to be done for a more robust solution. This is one of the reasons why I have provided the code "AS IS" rather than saying it could be used in production.

I would want to resolve all of these restrictions and limitations if I was to write the loader up as an application note.

[dkinzer]

(1) How can the firsttime variable be set so that it appears that the firmware has been updated through traditional means?

Here is some example code that sets the "first time" flag:

Code: Select all

Dim firstTimeFlag as Persistent Boolean Based Register.FirstTime.DataAddress
Sub Main()
   firstTimeFlag = True
End Sub

Although it appears somewhat complicated, the beauty of doing it this way is that you don't need to know the address of the flag in Persistent Memory.

Note that Register.FirstTime is not in the current documentation, a fact that was discovered in the process of preparing this reply. It is a read-only variable that can be read without changing its state. This is in contrast to invoking the FirstTime() function which will return the current state but after the invocation the flag will be cleared.

2) Are there other variables in persistent memory that might be relevant?

The size, in bytes, and the CRC of the code last downloaded are stored in Persistent Memory addresses 8 and 10 respectively. There is currently no way to access these values other than using GetPersistent()/PutPersistent() or by using the based Persistent technique illustrated above.

You'll find the code size and CRC, both 16-bit values, in the .zxb file in the proprietary "type 56" record, an example of which is shown below. As you may know, the .zxb file contains "Intel Hex Format" records but we've added some special record types to suit our needs. The "type 56" record is a Persistent Memory data record.

Code: Select all

:0400085648009bb308

The other record types that we've added are "type 58" and "type 59". These are used for conveying the the minimum VM version number required for the code and the ZX device type, respectively. Examples of those two records are shown below.

Code: Select all

:050000595a583234008a
:030000580105059a

The data portion of the "type 58" record contains three bytes giving the minimum required VM version number. The example record above indicates that the minimum VM version number is v1.5.5.

The data portion of the "type 59" record contains an identification string indicating the device for which the code was compiled. The example above contains the device identifier "ZX24". Note that the device identification string is null terminated.

[abarnes]

I wouldn't call 8 hours work .. an afterthought.

I would! I thought it actually took longer... of course with your aggregate experience between BasicX and then ZBasic over the years, 8 hours of work in your time is probably equivalent to at least 40+ hours of work for most others...

In my experience the VM is rock solid.

My colleagues have reported the same. So no rush on an autoupdating VM feature, that would probably be overkill considering the track record.

However, some of the devices we provide have some exotic requirements and are depended upon for some considerably unusual applications which literally prohibit ever physically contacting the device again once deployed. The cost of recovering the devices physically, in many cases, would be considered prohibitive. If a very unusual bug were discovered (we put these chips under some pretty demanding conditions), our clients use of these devices could be impaired without a means for recovery.

We've built alot of redundancy into our hardware for this reason, though size is a very relevant concern also, so we must balance these requirements.

This is the primary motivation for the firmware remote-updating requirement on our functional spec. We trust the VM more than our own code, and we also want to be able to update our own code and add features remotely as needed.

In summary, a VM remote updater is desirable, even though it has an established track record -- considering our special requirements. However, considering the complexity of developing or contracting the development of a VM remote updater, we have decided to place less importance on this as a calculated risk.

As noted in my restrictions, I do not set the device EEPROM.

Yeah I saw that. Hence the question; we're going to be implementing a production-level remote bootloader. Damn I'm eager to start coding, but we're still about a week from officially starting that milestone.

(1) How can the firsttime variable be set so that it appears that the firmware has been updated through traditional means?

This data is held in the first 32 of the device EEPROM. It would be easy to set using the PutPersistent routine.

Hence the rest of my question...

In other words, what's the memory location of the underlying variable, and what are its possible values? Within the bytecode it appears that this is a proprietary VM call, which shields the memory address information through indirection.

I suppose one option would be to poll all of the memory before and after calling firsttime() from within running firmware and doing a diff, but that sounds rather tedious.

---

You'll find the code size and CRC in the .zxb file in the proprietary "type 56" record [...]

Yeah I noticed these unique records when I was looking over the Intel Hex spec. Then subsequently when I tried to read in the file using file readers w/ the Intel Hex spec. I could read them in and edit their contents using prebuilt readers & in-place editors, but only after I removed these "offending" records. I think I might tear into an opensource Intel Hex reader and place a patch for these records, though simply removing them and readding them isn't much trouble.

It's useful to know their defined meaning, thanks for the inside info.

the beauty of doing it this way is that you don't need to know the address of the flag in Persistent Memory.

I love that your platform supports this. That is elegant.

Note that Register.FirstTime is not in the current documentation,

Yeah, I noticed. I'm glad to know that there is already a reference available, so that the DataAddress property can be used. I do not believe the CodeAddress property is documented either, though I do not know if that was intentional.

Maybe a wiki would be a useful forum for documentation, and for post-documentation "clarification". Alternatively, the PHP guys actually have a standard HTML documentation format, but with a forum underneath each documented feature, where feature-specific questions are asked & answered, and often their documentation is updated based on the forum. It can be pretty useful, since often the coders can't anticipate all of the questions that might be generated by a particular feature.. and I imagine it saves the core developers time. Anyways, these are just suggestions.. the current documentation is quite adequate, especially in conjunction with this forum.

Don, thanks for the specificity of your answers. The ad-hoc documentation you just provided is very useful.

I might have some follow up questions on that later.

For now,
1) Does the VM do a CRC check on the firmware after ResetProcessor() or after a power source is removed and then reintroduced, or at some other point? Or does it only do a CRC check when the conventional zload process is used? I'm guessing it's the latter, since the former seems pretty unlikely.

2) If that is the case with the CRC value, then what might a motivation be for making sure this value is accurate after a remote update?

Thanks again, again,
Austin

[abarnes]

Looks like there is mention of it in the changelog for 2.2

"- Added Register.FirstTime (read-only)."

Though it obviously didn't make the pdf.