Hi @dlech ,

Coincidentally, this is a feature we've been internally discussing (in Gitter, so not really internally after all) as part of our "Extended Archives" feature (see the WIP manifest here: #707).

We've implemented configuration support in our Resin CLI (https://github.com/resin-io/resin-cli) already, but we're looking to extract this functionality and make it generic in Etcher.

This is what we currently have

• resin-image-fs: A very handy and reliable module to perform file operations in image's FAT partitions (like move, copy, replace, etc). It currently only supports FAT, but nothing prevents us from adding support for other file systems as long as its feasible to do so in a cross-platform fashion. We should also remove the "Resin" piece of the module, as it suggests the module is Resin.io-only.
• A way to define "questions" to ask the user, for example:

options: [
  message: 'Processor'
  name: 'processorType'
  type: 'list'
  choices: [ 'Z7010', 'Z7020' ]
,
  message: 'Coprocessor cores'
  name: 'coprocessorCore'
  type: 'list'
  choices: [ '16', '64' ]
]

• The answers from the options presented to the user are passed to resin-device-operations, a module used by the Resin CLI which accepts a set of configuration "commands" and executes them on an image. Check the following commands as an example:

[
    command: 'copy'
    from:
        partition:
            primary: 1
        path: '/bitstreams/parallella_e16_headless_gpiose_7010.bit.bin'
    to:
        partition:
            primary: 1
        path: '/parallella.bit.bin'
    when:
        coprocessorCore: '16'
        processorType: 'Z7010'
,
    command: 'copy'
    from:
        partition:
            primary: 1
        path: '/bistreams/parallella_e16_headless_gpiose_7020.bit.bin'
    to:
        partition:
            primary: 1
        path: '/parallella.bit.bin'
    when:
        coprocessorCore: '16'
        processorType: 'Z7020'
]

The actual commands make use of resin-image-fs.

This is what we're missing

• Polishing the way options and commands are declared. We want to ditch the concept of "commands" and use a more "high-level" way of declaring what a file goes somewhere, etc; since the current approach is very imperative.
• Add support for common "schema-like" languages, like JSON, YAML, INI, etc; and allow users to define that a certain value goes in a certain "key" of a certain file. We manage the rest.
• Maybe templating support?

In any case, this is all taking shape and we don't have anything clear at the moment, but if you're interested, we're discussing a lot of this stuff in Gitter, so I suggest you to keep an eye there.

Given that you opened this issue, lets make this the home of more organized discussion about this topic.

+1 Another usecase for such a feature would be to setup wifi. On many devices entering the wifi settings is hard. I don't want to complicate the feature request, but I wonder if it should be considered to be able to re-run the customization steps on a preflashed image?

@ensonic you'll be glad to hear that @jviotti is going to be spending the next two weeks focusing on this precise problem. Wifi config is certainly a big use case.

I dunno how suitable it is for Etcher, but you might want to have a look at http://www.pibakery.org/ ( https://github.com/davidferguson/pibakery - also based around NodeJS and Electron I believe) which was created to solve issues very similar to those being discussed in this thread :-)
https://www.raspberrypi.org/blog/pibakery/ ping @davidferguson

Hi @lurch ,

We've indeed looked at PiBakery, which is in fact based on Etcher (it uses Etcher's backend modules, and our elevation system, among other stuff).

While PiBakery is very Raspberry Pi specific, we're creating a completely generic system that not only allows users to configure, but to re-configure as well (read config settings back from the device).

We are currently working on this system, which is the reason why I'm not very active anymore (I'll get back to normal speed once the foundations are done), and hopefully everything will make more sense afterwards (we'll publish specs, etc).

We've indeed looked at PiBakery, which is in fact based on Etcher (it uses Etcher's backend modules, and our elevation system, among other stuff).

Oooh, that's interesting! In that case I wonder why @davidferguson uses his own CommandLineDiskImager program (forked from Win32DiskImager), rather than using the image-writing code from Etcher?

we're creating a completely generic system that not only allows users to configure, but to re-configure as well (read config settings back from the device).

Sounds cool. If you've not already done so, you might want to have a look at Webmin and LuCI for ideas / inspiration / sample-code :-)

In that case I wonder why @davidferguson uses his own CommandLineDiskImager program (forked from Win32DiskImager), rather than using the image-writing code from Etcher?

I use CommandLineDiskImager for Windows because I was getting some really weird errors when I tried to use resin-image-write on Windows (can't remember the exact error now, but it appeared in resin-image-write, etcher-cli and also the main Etcher app, and on multiple Windows computers) so I decided to use Etcher on Mac (and Linux) and a version of Win32DiskImager on Windows.

I'll try resin-image-write again on Windows again at some point, but the error was tricky to reproduce and seemed to appear at random times, so for the moment I'll probably stick with CommandLineDiskImager.

@davidferguson you're probably referring to the issue narrated in this post -- https://resin.io/blog/the-perils-of-writing-disk-images-on-windows/ , which have now been resolved in the latest version of etcher-image-write. The resin-image-write module is deprecated and we generally recommend people use etcher-image-write instead, which should be a drop-in replacement.

@alexandrosm That's the error - thanks for linking that! Great news that the issue is fixed - I'll work on using etcher-image-write in the next release of PiBakery.

In the spirit of opening up the development of Etcher further, I'll paste here a high-level document of the discussions we've had with @jviotti on the configuration pipeline. It's super abstract and ambitious, but we think we've actually cracked it. Juanchi has been on this for the last week or so, and by the end of this one he should have a version zero for another resin.io project, which should be easy to transfer to Etcher shortly thereafter.

Device (re)configuration for Etcher

Introduction

Users that write OS images with Etcher, often want to configure these images with Etcher as well, before they boot the devices that these images are intended for. In addition, they want to be able to reconfigure these images later, either because they may have made a mistake, or because the settings are no longer correct and the device does not offer a way to change its configuration, very common with headless IoT devices.

Addressing this in a generic fashion is an extremely challenging problem, as the configuration information is often stored on the device in an unpredictable way, split between several files, each of which is of different format. To make things worse, these configuration files often do not express the user’s intent directly, but as a composite of various settings, which in aggregate express the user’s initial high-level input. Solving the configuration problem requires Etcher to have the ability to take user-level configuration and convert it into machine-level configuration, which is complex but possible. However solving the re-configuration problem optimally requires being able to reverse one’s steps, and convert machine-level configuration into user-level configuration, which is considerably harder.

If one is able to dictate the format of the configuration files on the device, things are fairly simple, as we can push the complexity to the device runtime. That’s how we do it on resin-cli for resin-os devices. However solving the problem in a general fashion requires being able to express as many different configuration approaches as possible.

This document describes an approach that excludes a fairly small amount of potential approaches to configuration, while at the same time being very economical with the metadata required to encode how to mediate between the user and the machine.

The four stages

The cornerstone of the approach is the definition of four distinct stages that the information goes through.

The first stage is the “user interface” stage, which is the stage in which the user interacts with the information. This may take the form of a web form, or a command line wizard, but the key is that it optimises for communicating with the end user and empowering them to express their intent, while avoiding errors, whether those are by misunderstanding or accident.

The second stage is the “dry json” stage. In this stage, the information is expressed as a JSON structure that is optimised for avoidance of repetition and internal consistency. This stage very closely resembles the visual stage, but is stripped of user interface considerations and reduced to a format that is easy for machines to process and transmit.

The next stage is the “wet json” stage. In this stage, the data is still encoded as a JSON structure, but instead of optimising for clarity and consistency, it optimises for matching the structure with which it is going to be stored on the device.

The final stage is the “file” stage, during which the information is stored on the device itself, potentially in different files, and potentially with those files having different serialisation formats between them, and even several formats co-existing nested within the same file. This stage is optimised for consumption by a running operating system, and while users are sometimes told to edit these files directly, the process is often quite technical and error prone.

In order for our approach to be fully described, we must examine each pair of stages, and how information is converted from the one to the other.

Transformation pipeline

In order to define a fully two-way transformation, we need to define two-way transitions between each pair of stages. Our pipeline should look something like this:

user interface <--(transition)--> dry json <--(transition)--> wet json <--(transition)--> files

Stage transitions

From user interface to dry json

The first transition is pretty straightforward, if only because this problem has been solved well by various existing systems. One system that is a good example, regardless of whether we decide to use it ultimately, is http://schemaform.io/. It can generate HTML forms from JSON Schema files. Those forms can be used to extract information from a user, which they convert to JSON. In reverse, given the data in JSON form, they can use it to populate a form, therefore enabling full two-way transitions. We may not use this transition as-is as we want more datatypes than just what is available in json schema, but the main approach will not diverge much.

From dry to wet json

This transition is the hardest one, and the one that is the least obvious. A single-field entry in the dry json form can become several fields in the wet json form. Further, those several fields can, in fact, operate as templates on which other fields from the dry json structure must be mapped.

While it’s not hard to think how to go from dry to wet json with the use of a templating system such as handlebars, it’s much harder to think how to reverse the process without having explicit code with which to do the transformation. The extremely interesting finding here is a library called jsonexp, which can take patterns as inputs, and not only match them on an arbitrary JSON file but in fact, extract undefined subsegments of those patterns and return them to the caller. The exciting piece of information is that the “patterns” that jsonexp would need to convert wet json to dry json are the same that a templating engine such as handlebars would need to convert dry json to wet json, if one ignores small syntactic differences which are easily overcome. As such, these patterns can be used to concisely express the relationship between dry and wet json and be used by different tools to transition data in either direction.

Upon further examination, we’ve decided not to use jsonexp directly, as we would want to stay within the confines of json tooling, but we will be following a fairly similar approach.

From wet json to files

The final transition is that between wet json and configuration files. Since wet json stores information in the same way as the files themselves, all that is needed is to express which files are to act as stores of data, and how the data is to be serialised (selected from a set of understood configuration file formats). The information intended for this stage should be expressed in a way that allows for the transition to be done in the reverse direction as well.

Additional notes

A note on manual file alterations

For this entire system to work, we must account for the case where users alter the configuration files on a device, and the resulting files are outside the range of the patterns. In that case, the system must partially break the abstraction and fall back to allowing the user to manually edit the unmatchable segment. Ideally, the user will be able to restore the segment to a default value, and also unaffected segments of the configuration file (or other configuration files) will still surface in structured inputs.

As a corollary functionality, we must consider the case when the user wants to manually edit the target files, perhaps to create a configuration which is not available by manipulating the high-level configuration. This too must be made possible by including a “custom value” alternative on the appropriate input.

A note on reconfiguration

For Etcher to be able to reconfigure a device, it must have access to the transformation description files. As such, those should be written to a partition of the flash drive as defined by the extended archive, so that they can be re-discovered by Etcher when it discovers the drive in the future.

A note on raw files

There exist situations in which we want to transfer a raw file from the UI all the way to the disk. For instance, if a picture of a logo, or some binary executable such as a kernel module or .dtb is needed. These assets can either be kept as they are from the visual to the file stage, or converted to ASCII data which is encoded in the transitional dry and wet json structures.

A note on “filesets”

When a variable number of files have to be created, which correspond to the instances of a multi-member entity (think network connections a device should try), the concept of mapping to a single file may break down. It is possible however to think of a single “fileset” entity, each member of which is serialised in a given way (e.g. yaml). We can then maintain the mechanisms that write to a given entity, which in this case is a fileset, but this is reflected on disk as a collection of files, achieving our goal. Given the right selection rule (e.g. all the files in a directory, or all filenames matching a certain pattern), the process can be reversed and the fileset generated based on the raw files.

After a conversation with @alexandrosm regarding how (and if) image configuration can fit to v01, I have some exploration. The idea is to fit image configuration within 'Image Info modal'. This window can appear by user's request (when clicking the image name or the 'i' icon on the main UI window), or automatically when a new image is selected (only when necessary).

Pinging @jviotti & @taahirisaacs, to initiate this discussion.
*Note: The modal is scrollable and current content is a rough estimation (would change according to the selected image).

Current 'Image Information Modal':

Potential Update:

I like where this is going. I think we should definitely show this by default, and even have some sane way UX-wise to actually enforce it, since there might be images that make no sense to leave un-configured.

Re-using the image details for this is a clever idea, however I wonder how obvious it is to users that clicking an informative bubble on the first step will trigger a configuration dialog. Maybe we can have another bubble with say a "cog" icon to open a different configuration modal?

Another idea: could we rename the "Flash!" button label to something like "Configure & Flash!" if we detect the image can be configured? In that case, "Looks Good" in the configuration modal would become "Flash".

I'd also help to define the different type of controls that could take place in the modal. By seeing the current sketch, I have several questions from a technical point of view:

• The "Persistent Logs" feature contains a danger icon. How do we present that in the JSON schema?
• Several things are presented in a grid during the "Ethernet" configuration phase. How do we determine if we should show in a grid, one below the other, etc? Do we use certain heuristics?

The possible controls here are (let me know if I'm missing any):

• Number input
• Text input
• Password input
• Text enumeration
• Number enumeration
• Boolean input (a checkbox)

Re-using the image details for this is a clever idea, however I wonder how obvious it is to users that clicking an informative bubble on the first step will trigger a configuration dialog. Maybe we can have another bubble with say a "cog" icon to open a different configuration modal?

This is my main worry. Putting all of this configuration in a "details" pane just feels wrong... it would be much more natural to have an obvious way of accessing this configuration. My initial thinking is along the lines of a cog icon as you said, but that feels too subtle. Maybe it should pop up automatically when an image is selected.

It would also be nice if there were a way to see a quick summary of the chosen options; maybe everything that is non-default would be displayed in the image details along with the path.

I am trying to reduce the amount of total modals and group information when possible. I'll explore it a bit more and I'll come back with a new UI.

I think we should definitely show this by default

How on earth would that work? IMHO this should only be available if the image-metadata explicitly asks for it, because it doesn't make sense to try and "guess" what settings an arbitrary image might need, what format they should be stored in, which files / partition they need to be saved on, etc.

Going back to my NOOBS experience: In NOOBS there's a mini "configuration step" while the OS is getting installed, where the OS gets an opportunity to configure itself based on what partitions it's actually getting installed onto. Since NOOBS can't know how different OSes need configuring, it looks for a partition_setup.sh script (see e.g. this one ), and if found it simply passes it a list of partitions as command-line arguments. It's then up to the script to mount whichever partitions it needs to write to, do whatever configuration is necessary, and unmount the partitions again.

When I saw @konmouz 's design pictures I assumed the image-metadata would simply specify whether it wants the "ethernet config" and "wifi config" configuration tabs visible (and we could define other standard configuration tabs). But from what @jviotti is saying with his discussion of widgets and layouts, it sounds like he's imagining each extended archive will be able to specify completely customisable configuration UIs? (which is obviously another whole level of complexity)
If we are allowing custom-interfaces, then perhaps things like an "IP address input" control would also be useful? (or maybe just having a validation-regex option on the "Text input" (which the publisher could set to ^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$, or even the regex from http://www.regextester.com/22 ), would be sufficient?)
Are the tickboxes next to the textboxes in @konmouz 's pictures so that we can tell the difference between NULL (i.e. don't store this setting) and "" (i.e. store this setting, but set it to an empty string) ?

I assume we'll be using reconfix-configurations stored in the image-metadata to convert the configuration captured by Etcher into the OS-specific config files? And then using LKL (as discussed in Athens) to actually write those config files to the SD card? Or for this first version are we just going to be writing a single JSON config file to a nominated FAT partition on the card, and leave it up to the OS to make use of that config file however it chooses?
I guess (as an advanced option) we could even have a button to save the configuration as a 'default config' back into the image metadata itself (but just in the metadata, leaving the raw-image untouched), so that if the user is repeatedly writing images with the same config values (e.g. wifi login details) to multiple cards, they don't have to keep re-entering them each time they start Etcher.

Just having re-read through @alexandrosm 's "Device (re)configuration for Etcher" comment, one part left out of that which has just occurred to me is that if we're writing configuration files directly, we probably also need to be careful about the ownerships and permissions that new files get created with. (should reconfix just be given numeric uids and gids, or should reconfix be given the "more friendly" usernames and groupnames and attempt to read /etc/passwd and /etc/group/ from the image to lookup the appropriate uids and gids?)

Something else I've just thought of - obviously to ensure that we can verify the image got flashed correctly (by comparing checksums), we'll only want to "save" the config onto the SD card after we've flashed and verified it. But as https://resin.io/blog/the-perils-of-writing-disk-images-on-windows/ explains, on Windows we actually write the MBR-chunk of the image to the SD card last, because apparently Windows doesn't allow raw-access to 'partitioned' space. So in order to be able to write the config to the card, does that mean we'd actually have to read back the MBR-chunk from the card, erase the MBR-chunk, write the settings, and then write back the MBR chunk? :-S (and I wonder if that might lead to a catch-22 situation, with LKL needing to be able to read the MBR?)

How on earth would that work? IMHO this should only be available if the image-metadata explicitly asks for it, because it doesn't make sense to try and "guess" what settings an arbitrary image might need, what format they should be stored in, which files / partition they need to be saved on, etc.

Of course. This would only apply if the image has Etcher-readable metadata and specifies things to configure.

When I saw @konmouz 's design pictures I assumed the image-metadata would simply specify whether it wants the "ethernet config" and "wifi config" configuration tabs visible (and we could define other standard configuration tabs). But from what @jviotti is saying with his discussion of widgets and layouts, it sounds like he's imagining each extended archive will be able to specify completely customisable configuration UIs? (which is obviously another whole level of complexity)

I think this feature is pretty much useless if you can't configure the controls that are shown. That will definitely be difficult and design-intensive, as with the rest of this idea.

But from what @jviotti is saying with his discussion of widgets and layouts, it sounds like he's imagining each extended archive will be able to specify completely customisable configuration UIs? (which is obviously another whole level of complexity)

Exactly, what is shown is completely generated from the Reconfix schemas, and its indeed a big design challenge.

If we are allowing custom-interfaces, then perhaps things like an "IP address input" control would also be useful? (or maybe just having a validation-regex option on the "Text input" (which the publisher could set to ^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}$, or even the regex from http://www.regextester.com/22 ), would be sufficient?)

That is a great idea: to have a collection of more complex domain-specific inputs like IP addresses. I'll create an issue in Reconfix to track it.

I assume we'll be using reconfix-configurations stored in the image-metadata to convert the configuration captured by Etcher into the OS-specific config files? And then using LKL (as discussed in Athens) to actually write those config files to the SD card? Or for this first version are we just going to be writing a single JSON config file to a nominated FAT partition on the card, and leave it up to the OS to make use of that config file however it chooses?

Reconfix will do the whole thing. LKL is only needed when writing to partitions other than FAT32, but I think we're fine with that, and therefore we'll delay the introduction of LKL here.

Just having re-read through @alexandrosm 's "Device (re)configuration for Etcher" comment, one part left out of that which has just occurred to me is that if we're writing configuration files directly, we probably also need to be careful about the ownerships and permissions that new files get created with.

This is a very good point. I'll create an issue on Reconfix for it.

Something else I've just thought of - obviously to ensure that we can verify the image got flashed correctly (by comparing checksums), we'll only want to "save" the config onto the SD card after we've flashed and verified it.

Exactly, configuration will happen after the flash/validation process.

But as https://resin.io/blog/the-perils-of-writing-disk-images-on-windows/ explains, on Windows we actually write the MBR-chunk of the image to the SD card last, because apparently Windows doesn't allow raw-access to 'partitioned' space. So in order to be able to write the config to the card, does that mean we'd actually have to read back the MBR-chunk from the card, erase the MBR-chunk, write the settings, and then write back the MBR chunk? :-S

The problem described in the blog post only applies when freely writing bites using raw I/O on the drive. Reconfix, on the other side, will mount the filesystems and interact with them instead.

Any update on this?

We have a small team converting the proof of concept into a real module. Keep an eye on this repo: https://github.com/resin-io/reconfix/

I feel like this thread turned into feature creep insanity. Why not just allow the user to provide a "post-etching" script? Etcher would provide disk partition info to the script via environment variable.

Simple. Not specific to any hardware/OS/project.

As an iterative update to that feature, grep the provided script for a line that includes ".*expected environment variables:". If found, after the colon will be considered a JSON map of {"variable_name": "default value"} pairs that will be displayed in the UI with text fields.

Again, simple. Not specific to assumptions about wifi setup, etc. Etcher is a globally useful tool for imaging SD cards. It shouldn't become PiBakery.

I found this issue thread via another by @KM4YRI (via @lurch) that asked about a GUI WiFi config during the flash process for Etcher.

Just wondering, what's the latest state of this, and is it possible to add anything to a custom Raspbian OS image to flag to Etcher to ask for WiFi credentials in the GUI as yet?

#718 (comment)

And here is a stab at how you might define this …

Is there an intention to write a passphrase (do I misunderstand)?

If so: instead, aim to write a PSK; https://man.freebsd.org/wpa_passphrase(8) for example.

It's a "competitor product"[1], but Raspberry Pi Imager now supports some configuration options (albeit only for Raspberry Pi OS).

(feel free to delete this comment @zvin or @thundron if it's inappropriate)

[1] Well not really a competitor, as both Balena Etcher and Raspberry Pi Imager are free downloads, and both are open-source 😃

Is this feature still in the works at all?

It would be great to see this addition for at least a few of the most common images flashed using etcher. How can I help?

Is this feature still in the works at all?

It would be great to see this addition for at least a few of the most common images flashed using etcher. How can I help?

Write an acceptable proposal. I wrote mine above and it got no attention.

You could just code the feature, but if you got have a prior statement of approval from the maintainers, it may not be accepted. That’s how Free Software works.

Here's one possible idea:

• We are using web tech, that makes JSON Schema a natural fit, since there are good tools to work with it already

• To use a schema, you just give it a special filename and put it either in the same folder as the image, or in the zip with it.

• Configurations are also savable, and they also go in the same place, so you can ship a set of sample configurations.

• Keep things simple and have a pretty fixed structure of pages with different properties.

• The generated config file goes into the image that gets written

• In addition, it can also modify files via postprcessing scripts

• postprcessing scripts are JS (Or python, we need a sandboxing framework anyway) files that have access to the libraries you include in the filters directory. This allows you to do things like password encryption in a very general way.

• They have access to SD card contents, your configuration, time/date/location and basic stuff like that, and the internet, if you
  check a box allowing this. Easy link-up for commercial IoT images that need to link up to an API!

• If internet access is given, they can make use of a cache folder for downloading large resources.

• They also may output status messages and reject the configuration, or produce build artifacts like logs, or keyfiles.

• All postprocessing scripts are absolutely required to be idempotent, and in this way a card can be reconfigured at any time.

• By standardizing config as JSON schema you discourage any non-back-up-able or non-scriptable ways of handling that kind of data.

This should take care of the whole issue of configuring an image very nicely, in a very general way.

We can extend the idea by combining with the discovery proposal to allow for remote configuration. Integrate SSH and MDNS discovery, and we should be able to automatically deploy those postprocess scripts and the sandbox to run them, push the new config, and totally reconfigure everything.

The final piece of the puzzle, then, is the very common need to configure things with user files. But, if we make our "configurations" into folders, we can have a sync folder. A schema file can define rsync style bindings between the sync folder and the remote, and what subfolders to make in the sync folder.

With some work, users could enter the password and be able to rsync to and from root-owned folders.

For desktop-like systems, backup and restore of the entire home folder, etc, and whatever else, could be a one-click operation, syncing to a local folder ready to back up however you want.

For signage, syncing your media would become very easy. Pulling logs and recordings off a device becomes something anyone can do in Windows.

We use a command line script to accomplish this kind of pull and push backup, but it's rather hacky and a consumer would... definitely not like that.

All of these features could be done separately of course, this just shows how it could all fit together.

I haven't seen this tread so I've post mine suggesting the same feature. The Raspberry Pi Imager have the option the add this configurations and works very fine, but is just limited to raspberry pi. I think this would be a great functionality addition to Balena Etcher.