ossia | Open Software System for Interactive Applications

Case study: [re]capture

Fri, 23 Dec 2022 00:00:00 +0000

Case study: scored air filtration in [re]capture

[re]capture (2022) is created by Alice Jarry and her team, consisting of Brice Ammar-Khodja, Jacqui Beaumont, Asa Perlman, Philippe Vandal, and Ariane Plante at Concordia University, where I am currently operating as a postdoctoral fellow.

This post presents a case study on the writing of a score with ossia score for such a complete artwork involving a wealth of interesting behaviours, sensors, actuators and protocols.

The [re]capture project

Exploring the concept of ‘filtration’ using biomaterials, plants, and electronic strategies, [re]capture is a dialogue between in situ atmospheric data capture devices and gallery installation. With air toxicity a growing issue, this ongoing project seeks to give shape to the microscopic invisibility and the macroscopic dimension of air pollution.

With the support of the Conseil des arts de Montréal, (@conseil.des.arts.de.montreal), the Fonds de recherche du Québec – Société et Culture (@fondsrecherchequebec) and the Concordia University Research Chair in Critical Practices in Materials and Materiality.

There are two main physical items involved in this artwork: air quality sensors (Nomad AirKits) which sense various atmospheric properties - temperature, CO² and pollution levels, etc., and the main installation which is a set of modules consisting of LEDs, fans and motors in a tubular circuit involving various materials, most importantly filtrating abaca and bioplastics. The general idea is that changes in sensor data will make the motors release dust in the tubular circuit ; fans will propel the dust inside it and LED lights accentuates the dramaturgy and allows the public to visualize the process.

To learn more about the artwork, head straight to the Materials Materiality research chair website

Prototyping and producing a score

The general idea for this score is still grounded in experimentation: the production time with the actual entire hardware set-up was a few days ; a big part of which was about learning how to play with the material and physical properties of the piece in order to make it look interesting.

This section will describe the setup, the general compositional process and the various features of score that were useful during the production work.

The goal is to leverage the external sensor data to activate the various air filtration modules inside the piece, by triggering motors and fans. For instance: when the CO² levels of a sensor reach some particular limit, something should happen ; the light intensity could be mapped to the average temperature of all the sensors ; etc. These mappings should evolve and vary over time to make the piece more dynamic.

Connecting ossia to the devices

Before doing anything, we have to focus on how score is going to interoperate with the external world for this piece: how are we going to get and process sensor data, and how are we going to control the stepper motors, LEDs, fans of the main installation.

There are as such two connections: inbound, from the sensors and outbound, to the serial port.

Input devices

The inbound connection is trivial: it is sufficient to create a new OSC device listening on the port on which the Nomad AirKits emit, and use the OSC learn functionality. The current experiment has three sets of sensors operating, simply identified as sensors:/<sensor number>/co2, etc.

Output devices

The outputs are controlled directly through a fairly simple serial protocol: the electronics side of the piece is divided in multiple boards. Each board has multiple controls. All the output modules are handled by this electronic board:

A Nomad AirKit is also pictured above: the tube with the small solar panel sheets.

Remember than score allows to communicate with arbitrary systems over a serial port thanks to the Serial device, which defines how the device tree maps to the actual communication protocol in QML / JS.

Here is the Serial device source code used for this work, which replaces a complex PureData patch.

import Ossia 1.0 as Ossia

Ossia.Serial {
    // The serial boards have a very low baud rate (56k). 
    // To not spam them too much, we force a coalescing 
    // of the values coming out of score so that there is 
    // at most one new value every 20 ms.    
    property real coalesce: 20

    // This is the implementation of the protocol our serial board expects: 
    // a b c \r \n
    // 
    // `a` is the board number byte
    // `b` is the control n° on that board
    // `c` the control value.
    // 
    // We use Uint8Array in order to send binary data.
    function frame(array)
    {
      let auint8 = new ArrayBuffer(5);
      let uint8 = new Uint8Array(auint8);
      uint8.set(array, 0);
      uint8[4] = '\r';
      uint8[5] = '\n';
      return auint8;
    }
    
    // Quick global mapping for our parameters.
    function mapValue(val, tp)
    {
        if(tp === 64)
        {
            // This is a bit of an emergency cheat we added to disable the stepper motors
            // in the central range as they were otherwise too noisy.
            if(val < (64 - 55)) return val;
            if(val > (64 + 55)) return val;
            return 64;
        }
        else
        {
            return val;
        }
    }

    // All our boards have the same parameters, so we refactor the parameter creation
    // with a simple utility function.
    function createNode(board)
    {
        // Each parameter is pretty similar so we can also refactor this.
        const createObj = (sensor, desc, def) => { return {
            name: sensor.toString(), 
            type: Ossia.Type.Int, 
            min: 0,
            max: 127,
            value: def,
            description: desc,
            request: (val) => frame([
                board + 128, 
                sensor, 
                mapValue(Math.max(Math.min(val.value, 127), 0), def)
            ])
        } };

        return { 
            name: board.toString(), 
            children: [
                  createObj(0, "stepper", 64)
                , createObj(5, "fet5", 0)
                , createObj(9, "fet9", 0)
            ] 
        };
    }

    // We have four boards, we define them here:
    function createTree() {
        let arr = [];
        for(let i = 1; i <= 4; i++)
          arr.push(createNode(i));
        return arr;
    }
}

Test cues

Sensors were not always available: some needed repairing, and their short battery life means that it was not possible to access live air pollution data at all times. To still be able to test changes in the score, a very simple pattern is to create a state (by double-clicking in the background of a scenario), drop the values of the cue from the explorer, set a trigger on it and mark it as enabled. It is also possible to make a small sub-scenario with a loop in order to have some automatic variation of our simulated sensor state.

Experimenting with mappings

Using the “local play” feature was instrumental for quick testing of individual mappings: at any point, it is possible to go inside a mapping sub-scenario and hit “Local play” (the “Play” button in orange, below): only this specific mapping will be executed, which allows to concentrate on making it look nice and interesting.

Organizing the score

For this score, we chose a triptych organisation. There are three “panels” in the score, which are going to loop over a hour and a half. Each gives a different perspective on the piece and involve different mappings with the sensors and actuators part of the piece.

The first part contains individual mappings from sensors to modules. The second part is mainly pre-written automations, which are noisified and whose playback speed is altered depending on the external conditions. The last part is mappings from the average state of all the sensors, to all the modules taken together.

Making it loop

As you may know, it is fairly easy to make a score loop forever to run in installation mode, like this: it is done with the graph link, created by dragging from the small red plus icon when selecting the last state of the score.

Score experiment: triggering a fan after a motor

One of the low-level scoring needs of the piece is that the fans should start blowing after the motors trigger in order to make sure that dust is indeed being blown in the tubes. Here’s a simple score example of how to do this:

Score experiment: Emergency behaviours

One of the various ideas that were experimented with during the prototyping phase was to have a nominal behaviour for the piece, and switch to an “emergency” mode if for instance some sensor data is detected to be too high for human safety. The score shall go back to its nominal mode whenever the atmospheric particle levels go back to safer values.

Here is an example of a small score that implements this behaviour. This leverages all the syntax elements of the ossia score visual language: triggers, conditions, loop transitions…

Mappings

Mapping a set of sensor to a module

In this piece, we have as you remember:

Sensors (Nomad AirKits). These are ESP32 boards which regularly send air quality and sensor data over OSC. The sensed parameters are: temperature, humidity, TVOC, CO², PM1, PM10, PM25.
Modules: a module is a part of the installation which contains a LED ribbon, a stepper motor, and a fan to blow the dust inside the tubes.

The first approach for scoring, inspired from previous prototypes, has been to define a mapping / sub-score that processes arbitrary input sensors and writes it to an arbitrary module. In order to do this, we used a custom OSC device for storing score variables.

The composition rule for this part was that each mapping could leverage two distinct parameters, abstractly named “p1” and “p2” and map them to the three outputs of the module, stepper, led intensity and fan.

For this, we simply created a dummy OSC device which is used as a variables holder.

Creating such a device was actually fairly simple as it is possible to create multiple addresses thanks to OSC pattern expressions: the pattern module/{A..J}/{fan,led,motor,p1,p2} will automatically create most of the addresses needed in one go when entering the “Add child” dialog.

Then, we created our modules:

Each module has a slightly (or very) different behaviour. Some have pre-written parts for dramatic effect, while others are purely function of the input parameters. They are defined as free-floating intervals.

For each, the first trigger is configured to start whenever the OSC address variables:/modules/(the module)/run is set to true. Notice that “auto-play” is set on the trigger: otherwise, it will never start!

The last trigger is configured to kill the module when the run variable is set to false:

Then, inside each modules, the addresses are read live from the defined variables and used from the various objects in the mapping. For instance, the module A’s LED output is configured like this:

At runtime, whenever the address stored in variables:/modules/A/led changes (for instance it could be the string serial:/2/9, serial:/1/9 or even serial:/*/9 to address them all), the mapping’s output will be redirected to the new parameter.

Finally, we have a small temporal composition of states which will change the value of these variables over time ; the score is organised per-module board, that is, the first line defines all the modules used by the module 1, etc.:

Each state will change the module used at a given point in time by a board ; it looks like this:

This of course calls for some evolutions to ossia score to make this kind of compositional system easier to define and write for: mainly, we are missing the ability to instantiate and remove a given module on the fly and use them as a template.

Scripting

The second part of the score uses another approach: the general shape of the parameter evolutions for every module is written manually, in order to try to emphasis a slightly more dramatic narrative that the one we are able to achieve with simple parameter mapping.

The LED automations are not directly used to control the LEDs ; before, they go through an expression mapper which adds progressively more noise both over time and depending on the external conditions detected by the sensors.

Finally, we also use an average humidity measure to control the speed at which these automations are being played, by using a tempo process and writing a small mapping. In many parts of the score, the Smooth object was extremely useful: the sensors only give updated values every 5-10 seconds on average, which would sometimes give very surprising jumps. Smoothing over a second-ish allows to avoid that.

Global mappings

Finally, the last part of our tryptich focuses on mapping averages of our sensor data into communicating behaviour across all the modules. For this, we used for instance the Sweeper object described below, which makes a specific behaviour “run” across all the serial modules in a rhythm defined by one of the sensor data. Multiple mappings succeed one another. Below, one is pictured: it maps the CO² average of our sensors to both light intensity and circulation speed of the LEDs and fans.

Video mapping

The piece also features a video projection of video recordings done at the sites where the Nomad Airkits did air measurements. The video feed, captured with a 360° fisheye camera, is re-projected on the artwork.

On squishiness and light adjustements

The video projection used made the video too wide: we used a very quick and dirty Video Mapper object to make it more round to fit the artwork. Likewise, some exposure adjust was needed as the video projector we had really wasn’t bright enough. Thankfully, ossia score’s user library comes with objects for handling this built-in, thanks to the great ISF library shared by Vidvox!

Mapping parameters to real-time video filters

In addition, we can of course use parameters from our sensors to adjust the video. For this experiment, we simply mapped the measured temperature to the chromaticity of the displayed picture ; more interesting mappings could very likely be devised over time.

New objects for pattern control

As part of the conception of this artwork, a few new objects made their way into score, which can be generally useful to the community. They could be developed very quickly thanks to the magical superpowers of Avendish which makes writing custom ossia objects an absolute breeze ; the development time for each object was at most one hour for each.

Here’s an overview:

Calibrator

This object was born from the observation that every sensor would have a fairly different range depending on the time of the day. It calibrates the input signal with its actual min and max bound over time, in order to enable to detect meaningful variations in the signal. It also provides some useful statistical analysis tools: mean, variance, lower and upper bound.

It is available under Control > Mappings in the processes pane.

Pattern combiner

This object fetches and processes addresses matching an OSC pattern, and allows to compute for instance the mean of the matched parameters, combine them in a list, etc.

Note that if you just need to get all the messages in succession from a set of OSC addresses, no particular object is needed: you can just directly use a pattern match expression as the input of a port in the address field.

It is available under Control > Data Processing in the processes pane.

Pattern applier

This object takes a list as input, and an address pattern for output which will match multiple addresses.

For instance: if the pattern is serial:/*/5, the actual addresses will be in our case serial:/1/5, serial:/2/5, serial:/3/5 , serial:/4/5.

The object will send every element of the list passed in input to the addresses, e.g. if the list [4, 6, 12, 7] is passed as input to the object, then serial:/1/5 will get 4, serial:/2/5 will get 6, serial:/3/5 will get 12, etc.

It is available under Control > Data Processing in the processes pane.

Note that if you need to send the same message to every address matching a pattern, no specific object is needed: OSC pattern match expressions can already be specified for every output port of score.

Pattern sweeper

This object sweeps over the addresses specified in the pattern expression at a given rhythm.

For instance, in the screenshot below, every quarter note, the next address matched by the pattern will get the input routed to it.

It is available under Control > Data Processing in the processes pane.

Conclusion

Thanks for reading all this! In the end, I must say that I had a lot of fun writing the score for this piece ; it also brought some small usability issues to the spotlight, which will be worked on in further releases.

Generalized "value" types and prototypal operation for C++ objects

Thu, 01 Dec 2022 00:00:00 +0000

This article is a followup to the series on interoperability and reflection. It covers extending our data model so that dataflow nodes can interoperate with not only basic types such as floats, doubles, etc. but also with more complex data models, in order to enable advanced data processing pipelines.

It also introduces a few new objects and change in ossia score that leverage these low-level improvements to enable to write nice data processing pipelines with a small reimplementation of the jq expression language.

Problem position

For instance, today one can write a simple Avendish plug-in with ports which are ints, floats, strings, bools, etc. This covers many use cases in media art systems. What happens when one wants to interoperate with, say, Javascript code which may return complex nested objects? How can we handle dataflow processors whose output is fundamentally non-trivial, such as a live feature recognizer outputting a list of objects recognized in a given camera frame:

[ 
  { type: "flower", rect: [0, 0, 20, 50] }, 
  { type: "person", rect: [100, 150, 200, 180] } 
]

TL;DR: The good news is that it now works, too! That is, an Avendish plug-in can now have the following port type which is a sink for recognized objects by some ML toolkit, e.g. a YOLOvX algorithm:

struct recognized_object {
  std::string type;
  std::array<float, 4> rect;
};

struct {
  struct {
    std::vector<recognized_object> value;
  } recognizer;
} inputs;

and any “compatible” input will be magically converted ; this means that even complex dataflow objects just need to speak roughly the same protocol to be interoperable, but not necessarily to use the exact same types, which allows to develop compatible libraries without requiring everyone to include gigantic base libraries such as OpenCV & the likes.

More precisely: where today one writing a PureData object would have to implement the “serialization / deserialization” protocol from PureData by hand by checking that the user’s types indeed match the types of the object’s internal data structures, now Avendish can automate that entirely.

There is a small friction point: many creative coding environments have a list type, but not all of them have a “dict” / “object” / “map” / associative container type. In particular, OSC (Open Sound Control, a fairly common network protocol in media art) itself is a sort-of-dictionary, but does not really have a way to represent dictionaries as OSC values.

Another one is that these environments generally have very, very, weak typing rule; many of our users in ossia expect the following to hold: 1.25 == "1.25".

Aggregate member name reflection

Sadly, C++ does not have enough reflection to enable us to retrieve field names from aggregates. Thus, by default, they will get converted into lists:

struct agg
{ int a = 123, b = 456; }

will be converted into [123, 456]. If one wants { a: 123, b: 456 } the field names have to be specified by hand until C++ evolves a bit:

struct agg
{ 
  int a = 123, b = 456; 
  static consteval auto field_names() { return std::array{"a", "b"}; }
};

Compatible types

Most types that conform to a subset of the std:: concepts for, say, containers, e.g. SequenceContainer or AssociativeContainer should work. In my tests I mainly tried std:: and boost::container::, please report issues with any other container library! Even funkier stuff such as std::bitset works :-)

static_assert(avnd::set_ish<std::set<int>>);
static_assert(avnd::set_ish<std::unordered_set<int>>);
static_assert(avnd::set_ish<boost::container::flat_set<int>>);
static_assert(avnd::map_ish<std::map<int, int>>);
static_assert(avnd::map_ish<std::unordered_map<int, int>>);
static_assert(avnd::map_ish<boost::container::flat_map<int, int>>);

Here are some complete test cases to get an idea of what works.

Value types?

The title of this article mentions “value” types. This is not about the common C++ concept (an int is a value type) but about types that are literally called “value”. You likely have encountered a few of those in various codebases. They tend to look like this:

using value = variant<double, string, list, dict>;

In older systems, the variant may have been implemented through ways other than templates, e.g. Qt’s QVariant, Poco’s Poco::Dynamic::Var, OpenFrameworks’s ofParameter::Value.

Here are a few excerpts of current codebases, to show the extent of the pattern:

mspraggs/loxx

using Value = Variant<double, bool, Object*>;

mrdepth/libdgmpp

using Value = variant<int, double, bool, std::string>;

mapnik/mapnik

using value_base = util::variant<value_null, value_bool, 
    value_integer, value_double, value_unicode_string>;

mgiseski/dvisvgm

using Value = mpark::variant<
	PDFNull, bool, int, double, PDFName, PDFStream, 
    PDFIndirectObject, PDFObjectRef, PDFOperator, 
    std::string, std::unique_ptr<PDFArray>, std::unique_ptr<PDFDict>
>;

openpower/hostboot

using Value =
    std::variant<bool, uint8_t, int16_t, uint16_t, int32_t, uint32_t, int64_t,
                 uint64_t, double, std::string, std::vector<uint8_t>>;

openbmc/phospho-net-ipmid

using Value = std::variant<bool, uint8_t, int16_t, uint16_t, int32_t, uint32_t, int64_t, uint64_t, double, std::string>;

dpacbach/parsco

using value =
  std::variant<number, string_val, boolean,
               std::unique_ptr<table>, std::unique_ptr<list>>;

opencomputeproject/HWMgmt-MegaRAC-OpenEdition

using Value = std::variant<bool, std::string>;

epoupon/lms

using Value = std::variant<std::string, bool, long long>;

etc etc… this is only from the first page of the Github code search ; to say that the pattern is pervasive is an understatement.

It’s of course also used in ossia, ossia::value is the central type of the entire system.

The sad thing is, all these types are pretty much always entirely incompatible between them. Something as simple as

std::variant<int, bool> a;
std::variant<bool, int> b;
a = b;

does not compile because even if a and b are semantically… pretty close, they are in fact entirely different types. So something such as:

std::variant<double, std::string> a;
std::variant<bool, int, float> b;
a = b;

still doing more-or-less meaningful conversions (one could define this as “conversions that loose as few bits of information as possible”) between the types of a and b is entirely uncharted territory at least from the point of view of the C++ language’s standard facilities :-)

This all leads to another reinventing-the-wheel drama such as the one we have already mentioned in previous articles: an algorithm that uses “value” type A will have to be rewritten to work with “value” type B, even if these types are compatible, unless one is ready to turn the entire code into generic code which causes its own set of issues.

Value type support in Avendish

Very simply put, this works too :-)

Here is a complete example test-case that we use: TestAggregate is the static type defined both as input and output of the object, simply to make sure that both input and output of arbitrary types work correctly.

struct Aggregate
{
  static consteval auto name() { return "Aggregate Input"; }
  static consteval auto c_name() { return "avnd_aggregate"; }
  static consteval auto uuid() { return "a66648f4-85d9-46dc-9a11-0b8dc700e1af"; }

  struct TestAggregate
  {
    int a, b;
    float c;
    struct
    {
      std::vector<float> a;
      std::string b;
      std::array<bool, 4> c;
    } sub;

    using variant_t = std::variant<int, std::string>;
    variant_t var;
    std::vector<variant_t> varlist;
    std::map<std::string, variant_t> varmap;
  };

  struct
  {
    struct
    {
      static consteval auto name() { return "In"; }
      TestAggregate value;
    } agg;
  } inputs;

  struct
  {
    struct
    {
      static consteval auto name() { return "Out"; }
      TestAggregate value;
    } agg;
  } outputs;

  void operator()() { outputs.agg.value = inputs.agg.value; }
};

A simple JS script can be used from score for testing that this works correctly:

import Score 1.0
Script {
  ValueOutlet { id: out1 }
  FloatSlider { id: sl; min: 10; max: 100; }

  tick: function(token, state) {
    out1.value = [
         1, 
         2, 
         sl.value, 
         [ 
           [ sl.value, 0.6 ], 
           "foo", 
           [ true, false, true, false ]
         ],
         "hiii",
         ["hello", 123, "foo", 456],
         { "test": 123, "another": "variant"}
    ];
  }
}

This gives us the expected output, without us having to write any explicit serialization / deserialization code, thanks to the magic powers of Boost.PFR and C++20 concepts:

jk: jq for any object

In order to manipulate this kind of data easily at run-time without having to resort to a full-blown very slow JS interpreter, I decided to reimplement a subset of an extremely useful tool for working with JSON-structured data: jq. The object in score is called Object Filter ; the meat of the expression language parsing is done in the jk repository.

One can appreciate that the actual parser fits in 75 lines of C++, thanks to the awesome superpowers of Boost.Spirit! The library also leverages C++ coroutines quite a bit, as they lend themselves very nicely to the kind of operations we want to do with the library, enable lazy computations, etc.

Primer of jq syntax

jq is able to process and transform JSON thanks to a very simple expression language. For instance, .foo[2:4] will iterate over the values 2 to 4 of the array foo in the following json:

{ "foo": [ 1, 3, 5, 7, 3, 5 ] }

Another example: given [ .foo[2:5][].bar ] and the JSON

{
    "foo": [
      { "bar": 1 },
      { "bar": 3 },
      { "bar": 4 },
      { "bar": 2 },
      { "bar": 5 },
      { "bar": 2 }
     ]
}

we are going to get as output: [ 4, 2, 5 ]. This can be tried on the online playground here. Thus, the tool is extremely efficient for manipulating complex JSON-like datasets - and our internal data model in ossia score is now compatible with this!

Only a small subset of jq has been implemented so far, but it’s already enough to perform a lot of useful data transformations. The best way to check if something is supported yet is to look into the unit tests… sorry not sorry :)

Implementation

Of course, jk needs a value type too. It is defined as follows and should cover most cases:

struct value;
using string_type = config::string;
using list_type = config::vector<value>;
using map_type = config::map<string_type, value>;
using variant = config::variant<int64_t, double, bool, string_type, list_type, map_type>;

The actual types are configurable by the application using the library, in order to be able to use, say, fast hash maps, better variants than the std:: one (in ossia we mostly migrated to Boost.Variant2 for instance) or containers with custom allocators easily.

Combined with what we mentioned before, this means that the inputs and outputs of our Avendish object are simply of type jk::value and everything gets converted automatically from and to our internal ossia::value (which cannot be changed for legacy reasons), while enabling others to use the library without the dependency on libossia. This will allow making the object work in other environments, e.g. PureData when there is some time (and enough people interested in it!).

Usage

The main thing to be aware of is how the jk object will output its values. If things are enclosed in a set of square brackets, e.g. [ .foo[0], .foo[2] ], then the output of the Object Filter in ossia scor’e will be a list. If it is without square brackets, the object in ossia score will output all the values one after each other as different messages.

Treating OSC nodes as objects

A big change in libossia recently has been to remove the pretty much unused CHAR type and replace it with a MAP type (that is, a map<string, value>). This allows us to transmit arbitrary maps / dicts / … between objects when processing a score, and more generally get very close of the JS object model which is fairly ubiquitous and well-understood nowadays.

The first use of this is to allow using a complete OSC node and not only a parameter as input of an ossia object: the children of the node will just be treated as map elements.

That is, given the following OSC tree:

OSC:/foo/bar 1.0
OSC:/foo/baz/a "hello" 
OSC:/foo/baz/b "bye"

if one sets OSC:/foo in ossia score 3.1.5 as input of an object, the object will receive a data structure akin to:

{
  bar: 1.0
  baz: {
    a: "hello",
    b: "bye"
  }
}

For instance, this allows to do things such as this to query specific members: in the following examples, the input of the Object Filter is set to Millumin:/millumin/layer.

Or do it more generally like this:

Or even recursively (in the next version as it was not implemented in time for 3.1.5):

Of course, when used as the input of a JavaScript object in score, one will get a proper JS object, which will enable much more advanced data processing!

Example

In this very simple example, we use a YOLOv4 recognizer’s raw output information to keep my side of the lab squarely into ǹ̸̙è̵̦g̷̹͒a̷̛̮t̴͚̿i̷̻̓v̴̗̎ē̶̜ ̶̩̐s̸̨̒p̴̨͛a̶̲̕c̷̢̿ë̷͈́. Now that the groundwork has been put to enable this kind of advanced objects to exist, they are going to slowly appear into further score versions.

Conclusion

Overall, the whole point of this is to allow us to transparently bridge between run-time-defined objects and transformations such as in JS or OSC, and compile-time static types to enable data processing in every way possible ; C++ objects for ossia should be able to define arbitrarily complex data structures for their inputs and outputs, and things should still be perfectly useable from the run-time environment without the end-user feeling restricted in what they can do, and with the object implementer still being able to write the cleanest code possible that solves the task at hand without having to worry about the interoperability glue code.

I think we are not far from succeeding :-) very likely a few other objects will have to be developed to simplify common use cases, but the general idea is there and held water pretty well over a few months of testing.

Score on the Store

Sun, 31 Jul 2022 00:00:00 +0000

Getting ossia score on the Microsoft Store

TL;DR: we released ossia score on the Microsoft Store, support and fund the development by buying it there!

ossia score is an interactive and intermedia sequencer: it allows to score multimedia content and processes in an interactive timeline, such as sound, MIDI, OSC messages, gamepad controls, video, VST plug-ins, scripts, with a specifically tailored visual language.

Technical details

What a drag this was! The Microsoft documentation is fairly complete, but also full of pretty old data – it is for instance not really clear how putting a Win32 app relates to UWP, to which version of UWP as the docs describes different ways between Windows 8 and Windows 10, etc.

But, in the end, we managed to get a reproducible procedure that I’m happily sharing here to allow other open-source software to be distributed there – it can be for instance mostly automated in Github Actions. In particular, it can likely be useful for anyone who uses a CMake / Qt / Clang toolchain (or any MinGW-ish toolchain) for building on Windows.

Prerequisites

Visual Studio installed with the Windows SDK of your choice.
An application that builds cleanly on Windows: running cmake --build . --target install should yield a clean installation folder with your app’s files.
A good few hours and roughly 3.5 liters of coffee and a couple liters of tea. Hopefully a bit less if you follow the steps below!

Steps

Account creation

First you need to register as a Microsoft Partner.

Fill in your banking details, tax forms, etc etc.
Create a new app, and set up the submission, description texts, etc etc.

Asset setup

A ton of different assets are required. Fortunately, it turns out that Visual Studio can generate them in the required format from a single base logo.

Open Visual Studio.
Create a new Windows UWP project, name it for instance MyAppAssets or something like this.
Click on the “Package.appxmanifest” on the file list on the left. This will open an UI editor for the manifest.
Set up the properties in the first tab, then switch to the second tab.
Set the source logo on the first field as explained here.
Hit Generate. The Assets folder in the source directory should now contain all the generated PNGs at the right scales ; you can move them back to your repo and close VS.

Of course, one could also generate them from a well-thought-out imagemagick script, if anyone has one please share it!

Package manifest setup

This is the manifest that describes the package for your software. The one we made can likely be used as a source. Note that this is a .in file to indicate that we are generating it from CMake (to set the version number from the git tag, mainly).

You must in particular fill in the various identity-related variables that are available on the Microsoft Partner account:

Home > Apps and games > MyApp > Product management: Product Identity

Here you’ll see variables such as:

Package/Identity/Name                    # e.g. "com.yourcompany.yourapp"
Package/Identity/Publisher               # e.g. "C=SOME LONG UUID-ISH THING"
Package/Properties/PublisherDisplayName  # e.g. "Your Company"

which are to be copied in Package.appxmanifest’s <Identity> and <Properties>, as well as a few other places mentioned later in this post.

Important Package.appxmanifest details for Win32 apps

The important steps for Win32 app validation to pass are:

EntryPoint="Windows.FullTrustApplication" in <Application>.
<rescap:Capability Name="runFullTrust" /> in <Capabilities>.

XML funsies

A few “fun” stuff I encountered while writing the XML:

The order of capabilities apparently matters: the <Capability ... /> (and more generally anything of the form <namespace:Capability ... />) must come before the <DeviceCapability ... /> ones in the <Capabilities> list.
The order of namespaces in IgnorableNamespaces="uap mp rescap uap2 uap3 rescap3" may or may not matter – I got weird errors with it.
Depending on which “version” of the appxmanifest XML namespaces we are using, some elements are sometimes namespaced, sometimes not. It would not be an issue if everything on the internet was up-to-date, but you can’t expect to copy some steps from a 2017 StackOverflow answer and have things just work :p

Package layout XML

This file is used to describe all the sub-packages that compose your software: it can allow to do elaborate things such as providing a separate AMD64 and ARM64 pacakges, plus a separate asset package for the architecture-independent data such as images, sounds, etc.

The most important thing to notice is ManifestPath=".\manifests\Package.appxmanifest".

This has to be the path to Package.appxmanifest, relative to the folder from which you are going to run the packaging procedure.

e.g. If you run the packaging procedure from %YOUR_INSTALL_PREFIX%\ the Package.appxmanifest.in file will have to be configured as %YOUR_INSTALL_PREFIX%\manifests\Package.appxmanifest.

Application manifest

Yay, more XML! You also need to make sure that your main application executable has a valid manifest associated, with explicit high-DPI support.

We generate ours from CMake as at some point there’s likely going to be a version number in there too.

The important steps are:

Setting dpiAware and dpiAwareness (which are what different versions of windows will use).
Indicate the supported Windows versions ; in our manifest for instance we support 7, 8, 8.1, 10 and 11. The UUIDs for various versions can be found here.

Generating the package

We now have all the data required for creating an .appxbundle. Apparently one can also generate a .msix / .msixbundle but, well, appx works!

Generating a certificate

The submission requires the code to be signed with a certificate, which matches the information provided by Microsoft:

Package/Identity/Name                    # e.g. "com.yourcompany.yourapp"
Package/Identity/Publisher               # e.g. "C=SOME LONG UUID-ISH THING"
Package/Properties/PublisherDisplayName  # e.g. "Your Company"

We can use Powershell to generate a self-signed one – thanks to OpenTTD for the example!

Here the subject and password are passed through Github secrets ; it’s not really necessary though as the certificate is regenerated for each build. From what I could read, Microsoft will then replace this certificate with their own once they validate the app and make it available to the store.

For instance, to generate a certificate for your own use on a dev machine, with the very strong password “toto”, it would look like:

$cert = New-SelfSignedCertificate `
  -Type Custom `
  -Subject "C=that-long-uuid-given-by-microsoft" `
  -KeyUsage DigitalSignature `
  -FriendlyName "friendly name given by microsoft" `
  -CertStoreLocation "Cert:\CurrentUser\My" `
  -TextExtension @("2.5.29.37={text}1.3.6.1.5.5.7.3.3", "2.5.29.19={text}")

Export-PfxCertificate `
  -cert "Cert:\CurrentUser\My\$($cert.Thumbprint)" `
  -FilePath c:\myapp\mycert.pfx `
  -Password toto

Be careful with rogue spaces, everything must be exactly as given on Microsoft’s website! Once done you should have a certificate file generated in c:\myapp\mycert.pfx.

Packaging

Start a Visual Studio 2022 Native x64 Command Prompt or whatever equivalent you have.
Go into your install folder (the one which contains your locally installed app, after make install or whatever).

In it you should have:

Your app’s .exe, for instance myapp.exe.
Your app’s .exe.manifest (which we generated above with CMake though this is of course not the only way).
Your app’s .DLLs, for instance Qt6Core.dll.
manifests/Package.appxmanifest

Then, we’ll follow the steps in that file.

Remove all unwanted files or files from a previous build / packaging step.
Create the resource files (here assuming a single language), used to indicate that e.g. all the SmallTile.scale-XXX.png files map to the SmallTile.png referenced in Package.appmanifest:

> makepri createconfig /cf priconfig.xml /dq en-US
> makepri new /pr %cd% /cf %cd%\priconfig.xml /mn %cd%\manifests\Package.appxmanifest

Embed the .exe.manifest in your .exe:

> mt -nologo -manifest myapp.exe.manifest -outputresource:"myapp.exe;#1"

Sign all the executables and DLLs with the generated certificate: for instance, if the password for c:\myapp\mycert.pfx was “toto” as above, it would look like

> signtool sign /fd sha256 /a /f c:\myapp\mycert.pfx /p toto myapp.exe
> signtool sign /fd sha256 /a /f c:\myapp\mycert.pfx /p toto Qt6Core.dll
> etc...

Build the package (finally!). The following command will put the resulting package in c:\output.

> makeappx build /v /f c:\path\to\PackagingLayout.xml /op c:\output /ca

Note that if you made any mistake in the Package.appxmanifest, this will fail and you’ll have to dig through the errors to understand what went wrong… good luck!

When you have a .appxbundle, you need to sign the package itself. The x64.appx generated next to it can be entirely ignored.

> signtool sign /fd sha256 /a /f c:\myapp\mycert.pfx /p toto c:\output\my-app-package.appxbundle

If this step fails, it is likely that the Package.appxmanifest file’s fields (Publisher=, etc) do not match the information used while generating the certificate.

If it works, congrats, we can move on to testing!

Testing the packaged app

One thing that bugged me the most was allowing the .appxbundle to install as it requires Windows to trust the generated certificate (which it won’t just by generating it).

Installing the certificate

I read a lot of StackOverflow answers and none did work – e.g. adding the certificate manually from the generated .pfx file into Local Machine > Third-Party Root Certification Authorities or Root CAs, etc etc. What worked was right-clicking on the .appxbundle and adding the certificate from there (to the same location, Local Machine > Third-Party Root Certification Authorities) instead of from the .pfx file.

Running the app

From there the app can install by double-clicking the .appxbundle, and you can finally try to run it from the Start menu! If it works, we are nearly there. If it crashes, well, time to inspect some coredumps :-)

Checking with Windows App Cert Kit

This utility validates that the package is well-formed and does not have basic issues. The same checks will also be run by Microsoft when you submit your app so it’s a good idea to run it locally as it will be much faster – Microsoft will reject your app if it does not pass. Simply pass it your .appxbundle to ensure that it passes.

Note that you may have errors show up in the “Optional” section if your program calls child processes, etc. It won’t make the submission fail even if “error” is written in a scary red but apparently it will prevent your app from being installed on restricted Windows versions.

Note that if you really want your app to work there and you use Qt, you will likely have to rebuild Qt by passing -no-feature-process to configure in order to disable the calls to CreateProcess that are in QProcess’s implementation, even if you do not use QProcess directly. Likewise for any other dependency you use which may have calls to LoadProcess and ShellExecute.

When it’s good, congrats, you can finally submit the package to the store and cross fingers for it to be quickly validated!

Collaborative and distributed interactive scores

Tue, 14 Jun 2022 00:00:00 +0000

Collaborative features in score 3.0.11

ossia score 3.0.11 introduces experimental collaborative features, that had been developed a few years ago as part of my thesis work, but had remained dormant since then (very precisely, since the presentation for the Journées d’Informatique Musicale 2017):

Recent projects involving distributed performance, such as João Svidzinski’s HERMES v2 distributed musical piece, an AFIM workgroup, a demo at SMC 2022, and a lot of great presentations at TENOR 2022, have reinvigorated the motivation for making this work correctly.

Two relatively orthogonal experimental features have been developed: rather than keeping them gated until it’s all perfect, we have decided to release them in a preliminary state as they can already be useful (with many caveats and disclaimers scattered through this blog post! No warranty, no stability guarantees, assume that everything can change in the next version!), and to gather feedback from live usage.

Collaborative edition: multiple people can connect together to edit a score just like in e.g. Framapad or Google Docs.
Distributed execution: the score can be annotated to indicate how the execution must occur across multiple machines and exchange live data between them.

For instance, it is easily possible to have a single score with synchronized playback, where a machine will process all the audio and another all the video, which is useful for complex scene situations requiring horsepower that a single computer would not be able to provide.

Caveat: live temporal synchronization is currently very barebones. Previous versions used to rely on proper clock sync at the system clock level, by using for instance PTP. Since this was developed before score even processed data and was backed by the audio clock rate, it has not been updated to make the synchronisation fit with the new execution pipeline. Further work on resyncing things correctly is planned !

Going network goblin mode

The architecture is client-server. A machine hosts, and the others connect to it. Everything is based on WebSockets.

The host can start hosting through the File > Make server menu option.
Clients can join a session with the File > Join menu option.

A network session allows both collaborative edition and distributed execution.

Caveat: if VST plug-ins and media such as sound files, videos, etc. are used, the paths must match on all computers, until we finish developing a proper distributed media management system.

Once a score instance is part of a network session, the “Network” panel will initialize itself:

There are three tabs:

Topology is about the definition of the network (more on this later).
Transport allows to play / stop everyone and control collaborative edition.
Object allows to set network-specific properties on objects of the score.

Collaborative edition features

The only “feature” of collaborative edition is that it can be optionally turned off.

This is useful for instance for playing with controls locally once a score has been defined and allows each client to have some amount of variation and play the collaborative score like an instrument.

In the following video for instance, the two score instances edit the parameters of a shader [1]; at any point they can “opt out” of sending new control changes.

Distributed execution

The interesting part of this work is the ability to specify, as part of a score, how it is going to be distributed across the network when executing.

That is, it is possible to say: “this part of the score runs on computer A”, “that part on computer B”. It is also possible to get the value of some computation or local input, and share it across the other computers.

A small abstraction level is introduced, to separate hardware from the high-level specification of the score: groups. In the “Topology” tab of the network panel, one can add groups:

The matrix below indicates which client is part of which group, a bit like UNIX users and groups. Clients are not saved with the score, but groups are.

Then, in the Objects tab, one can select objects of the score and assign a group to them.

For instance, in the video below, the top automation (A) is in group “audio” (there’s no audio, the name is just an example!), while the bottom automation (B) is in group “video”. The top client (“master”) is put in group “audio”, and the bottom client (“A client”) is put in group video. When executing, both scores process the ISF shader, but each only execute one of the two automations.

Distributing expression evaluation

The evaluation of triggers can be synchronized across the network, or left independent.

That is, one can choose whether a trigger is supposed to execute on all clients, or only on the client on which the trigger became true due to some external condition.

In this video, for instance, the top score instance is connected to a joystick, which triggers the successive automations. The bottom instance does not even have the joystick showing up in its Device Explorer - but when the topmost instance receives the trigger (a button press), the bottom one is informed and triggers too.

Caveat: synchronized triggering is only supported for triggers that are triggered through expressions involving devices, not yet when they are clicked with the mouse in the score.

Distributing interval speed

The interval speed can be shared too.

Both these sharing features are enabled / disabled at the scenario level, by indicating whether it is a “Free” or “Shared” scenario.

More fine-grained techniques are in a work-in-progress state in the codebase and not yet available for use, but the plan is to allow this choice for individual elements.

Execution data is local to each instance: by default, an LFO will be computed on each individual client for instance, and data is passed locally in the synchronous engine of the software.

A process has been implemented to allow sharing data across all instances, to allow for instance a live input on a client to be used to control an effect on another client… or even more fun things, such as allowing computing the mean value for a control for various clients, to enable laptop-orchestra-ish scenarios.

The process is called “Message pit”: it takes messages as input, and gives messages as output, by combining all the messages from all the clients which are sending messages to this process and executing it.

That is, given 3 clients A, B, C, if clients A and C send messages to a Message Pit process, and A, B, C execute it, then they will all see a message which is a combination of the two A, C inputs.

Multiple combinations are possible. Assuming that the last message sent by “A” was “1”, and the last message sent by “C” was “2”

List: A, B, C will see the list [1, 2] or [2, 1] in output of the process.
Average: A, B, C will see the value 1.5 at the output of the process.
Sum: A, B, C will see the value 3 at the output of the process.

Similar support for audio, video, and MIDI is planned but understandably quite a bit more involved to do properly.

It even works on the WASM build

… but it crashes very quickly and is not what anyone would call useable at this time… yet this is where we are going! Interactive distributed scores spanning from microcontrollers to web pages ;p

it actually works 🥳 one score in a browser compiled as #webassembly, another as a desktop app, both edit and play the same document. sync is imperfect, wasm clicks, but I think that this opens up cool new avenues for distributed music performance, laptop orchestras, etc ! pic.x.com/Poh6jj1txf
— jm (@jcelerie) June 11, 2022

[1] Shader Based on “Twist” Shadertoy by fb39ca4 - https://www.shadertoy.com/view/XsXXDH ; Inspired by Matthew DiVito’s gifs https://cargocollective.com/matthewdivito/Animated-Gifs-02 ; Credit: Joseph Fiola, Matthew DiVito, Shadertoy user fb39ca4”

Minimum viable declarative GUI in C++

Tue, 22 Mar 2022 00:00:00 +0000

Introducing a “toolkit”

Here is the code of a minimal GUI toolkit for defining declarative UIs:

// That's it. That's the toolkit.

Well, this plus some extremely routine parts of “ISO/IEC JTC1 SC22 WG21 N 4860 - Programming Languages : C++”, such as… strings… variable declarations… struct definitions… enums… ints… and a couple others. Code so post-modern it circled back to orthodoxy.

So, here is how one uses the C++ language plus this, uh, toolkit to declare a simple item, say, a text label:

struct {
  const char* my_label = "foo";
} my_ui;

Believe me, dear reader, that is actually enough user code to be able to show the following UI:

By user code, I mean that all the “nice” UI work can be done in, say, foo.hpp:

#pragma once
constexpr struct {
  const char* my_label = "foo";
} my_ui;

without requiring any custom header. Some yet-unknown .cpp file will include foo.hpp, and, no matter what the content of my_ui is, will execute a nice user interface from it. Anyone will be able to create independent libraries which will render UIs according to whatever platform-specific intricacies there are, but the actual UI code will be entirely independent from anything: peak separation of concerns is achieved.

Advanced UI

The code above can be trivially extended to showing not one, but TWO labels:

struct {
  const char* my_label = "foo";
  const char* other_label = "bar";
} my_ui;

The result:

Now, one notices how the visual position of the labels in the UI has the good taste of matching the one in the code: the language provides us with a native primitive for ordering disjoint things, so why not just use it ?

Of course, just putting elements in a vertical row may not be sufficient for every UI on earth.

Let’s declare that our UI embraces horizontality instead, in the simplest way possible that I could find:

struct {
  enum { hbox }; 
  const char* my_label = "foo";
  const char* other_label = "bar";
} my_ui;

And we get:

Nested UI

We may want to imbricate multiple things:

struct {
  enum { hbox }; 

  struct {
    enum { vbox }; 
    const char* a = "foo";
    const char* b = "bar";
  } group1;
  
  struct {
    enum { vbox }; 
    const char* a = "quz";
    const char* b = "qux";
  } group2;
} my_ui;

Gives:

We may want to go a bit deeper:

The controls are pointers to simple structs which define a couple of metadatas in the way mentioned in the reflection blogpost.

A nice thing here is that unlike many frameworks, there’s not much memory to leak, as the very structure of C++ structs is used to define the hierarchical UI.

Of course, there has to be a rendering framework somewhere: the nice thing is that the UI code has no dependency on the actual framework. Instead, some separate glue code will transmute our declarative UI definition. Thus, which framework exactly mostly does not matter, for a large amount of potential UIs.

To give an example, the screenshots above were taken with a first prototype of renderer which generates QML / QtQuick code (and runs it).

Trivial UI backends

Here is the exact same UI specification, but rendered with Nuklear instead, which is a fairly different UI paradigm code-wise:

I have spent an hour on it without prior knowledge of the library and did not find how to do splitters and group boxes well with it, but I’m sure there is a way !

Meanwhile… t r e e s.

The binding from this kind of UI specification to Nuklear holds in a ~200 lines (fairly dirty) file.

Here is the core loop, for reference:

void recurseItem(const auto& item)
{
  avnd::for_each_field_ref(item, [this] (auto& child) {
    this->createItem(child);      
  });
}

template<typename Item>
void createItem(const Item& item)
{
  constexpr int child_count = boost::pfr::tuple_size_v<Item>;
  if constexpr(requires { item.spacing; })
  {
    nk_label(ctx, " ", NK_TEXT_LEFT);
  }
  if constexpr(requires { item.hbox; })
  {
    nk_layout_row_dynamic(ctx, row_height, child_count);
    recurseItem(item);
  }
  else if constexpr(requires { item.vbox; })
  {
    recurseItem(item);
  }
  else if constexpr(requires { item.split; })
  {
    if (nk_tree_push(ctx, NK_TREE_TAB, "Split", NK_MINIMIZED)) {
      recurseItem(item);
      nk_tree_pop(ctx);
    }
  }
  else if constexpr(requires { item.group; })
  {
    if (nk_tree_push(ctx, NK_TREE_TAB, c_str(Item::name()), NK_MINIMIZED)) {
      recurseItem(item);
      nk_tree_pop(ctx);
    }
  }
  else if constexpr(requires { item.tabs; })
  {
    createTabs(item);
  } 
  else
  {
    // Normal widget
    createWidget(item);
  }
}

Conclusion

Glory to the post-library era and to declarative, struct & enum-based specification !

The only remaining step is to integrate it with score’s internal widget set and there will finally be a way to specify audio and media plug-ins in a way entirely independent from any host app or framework, with zero-cost abstractions.

Here is the example plug-in showcased above, done in two ways:

First version really only uses zero dependencies to show the sausage-making https://github.com/celtera/avendish/blob/07beb43db484341addc121ac18727258000c44bd/examples/Ui.hpp.
Second version use some helpers macros: https://github.com/celtera/avendish/blob/main/examples/Helpers/Ui.hpp. I don’t find them super convincing, but the nice thing is that the “public API” can be prototyped without issues ; as many distinct versions can exist concurrently. In the end, they pretty much won’t exist in the binary as things can be made constexpr fairly extensively: the whole layout object can be constexpr for instance.

A few small additional things may be useful:

An API for loading images, for instance to render pretty SVG background à la VCVRack.
Maybe investigating reactive properties. I have some ideas for those, which may actually fit in these margins :-)

This work can be tried with the avendish library.

Introducing ossia score 3

Sat, 05 Feb 2022 00:00:00 +0000

Introducing ossia score 3

We are happy to announce that after a few months of beta-testing, a stable version of ossia score 3 can be downloaded.

It has two defining features: first, the signal processing being applied can evolve over time. The first half of a score can have a given set of effects applied on sound, video… , and the second part of the score, a completely different set of effects being applied. Second, the timeline can encode interactivity: it is possible to state that a part of the score will only execute when someone touches a control or does something on stage, while others are kept in check with the musical beat, and then resynchronize these multiple competing timelines afterwards.

Our gallery presents a few artworks that use score and/or libossia, to give a feel of what people do with it: installation and video art, light shows, modular music

To get started, you can follow the tutorials that come with the documentation if you prefer reading, and check our Youtube playlists if you prefer to follow videos.

New features and changes since v2 and v3 beta

The beta blog post covers most of the new features since 2.5.2, as well as the roadmap. In particular, the next few months will focus on fixing all the small quality-of-life issues and bugs that can be encountered right now – things such as missing copy-paste in some places, UI bugs, unclear menus, etc. A high priority is also put on complete support for Spout and Syphon for video artists and mappers.

JSFX support

One last major feature was added during the v3 beta cycle: support for JSFX plug-ins (used mainly by Reaper). This is thanks to the ysfx library developed by Jean Pierre Cimalando, which was fairly easy to integrate.

The user library does not ship any JSFX plug-ins yet, but libraries such as provided by Joep Vanlier, Geraint Luff or Justin Johnson have been tested to work by putting them in the user library folder (score will look for .jsfx files).

Official Raspberry Pi builds

A lot of work has been put in ensuring smooth experience with the Raspberry Pi build, which has now been promoted from experimental to officially supported. A plain ALSA audio back-end has been added to drive various audio shields with minimal performance overhead when score is the only software running on the system, which would generally be the case for art installations.

Contributing to ossia score

ossia is a free software project at its core, and the free software values we hold dearly.

As such, we are open to contributions from everyone interested in improving it: if you want to start contributing to score and don’t really know where to look, come to the chat and we will do our best to mentor you!

A huge thanks

A huge thanks to the public and private sponsors who believe in the project – in particular Epic and the Megagrant thanks to which we could afford working full-time on ossia. To all the occasional and regular contributors, to all the individuals who take the time to report issues when they hit a bug, to all the developers and testers of the ton of libraries we use (check out the About screen once in a while!) and to all the artists who use score to create great artworks: thank you so much for your time and work!

Discuss this post on Reddit !

Genuary 2022 with ossia score

Sat, 15 Jan 2022 00:00:00 +0000

Genuary

Genuary is an initiative about encouraging creation of generative art thanks to a new oblique-strategiesque prompt every day of January.

This year, I set myself the objective of doing the daily prompts with score, and use it as an opportunity to showcase a few features, find some bugs, and try to find what is lacking in terms of generative abilities.

Here are all the videos so far (which started on Vimeo and swiftly moved to Youtube when I ran out of space).

The scores are accessible on Github. Some may require specific VST, LV2 or JSFX plug-ins.

JAN.1 (credit: Michael Lowe)

Draw 10,000 of something.

JAN.2 (credit: Anna Lucia)

Dithering.

JAN.3 (credit: Lionel Radisson)

Space.

JAN.4 (credit: Alexis André)

The next next Fidenza.

JAN.5 (credit: Thomas Lin Pedersen)

Destroy a square.

JAN.6 (credit: Alex Naka)

Trade styles with a friend.

JAN.7 (credit: Roni Kaufman)

Sol LeWitt Wall Drawing.

JAN.8 (credit: Bruce Holmer)

Single curve only.

JAN.9 (credit: Lionel Radisson)

Architecture.

JAN.10 (credit: Roni Kaufman)

Machine learning, wrong answers only.

JAN.11 (credit: Lionel Radisson)

No computer.

JAN.12 (credit: Richard Vigniel)

Packing (squares, circles, any shape…)

JAN.13 (credit: Ben Kovach)

800x80.

JAN.14 (credit: Devi Parikh)

Something you’d never make.

JAN.15 (credit: Thomas Lin Pedersen)

Sand.

ossia score 3 beta

Tue, 17 Aug 2021 00:00:00 +0000

Introducing ossia score 3

ossia score is an interactive and intermedia sequencer: it allows to score multimedia content and processes in a timeline, such as sound, MIDI, OSC messages, video, VST plug-ins, scripts, with a specifically tailored visual language. It has two defining features: first, the signal processing being applied can evolve over time. The first half of a score can have a given set of effects applied on sound, video… , and the second part of the score, a completely different set of effects being applied. Second, the timeline can encode interactivity: it is possible to state that a part of the score will only execute when someone touches a control or does something on stage, while others are kept in check with the musical beat, and then resynchronize these multiple competing timelines afterwards.

Version 3 fulfills the long-term goal of intermedia score authoring of the ossia project, by allowing our timeline to combine state-of-the-art audio, control and video processing pipelines in a single integrated system.

Where the version 2 focused on integrating audio processing to the execution engine, version 3’s defining feature are video processing & playback support, which opens ossia score to the VJ world, and musical metrics support, which enables its use in works requiring tempo and beats.

The beta is freely downloadable. We aim for a final release in early autumn, when all the bugs are ironed out during the beta testing cycle.

This post will present the new major features, and then sketch the way to version 4.

The road since score 2.0

This release cycle has been exceptionally long ; thankfully, we are nearing the end:

The last stable version, 2.5.2, was released on August 26, 2019, almost two years ago.
The first 3.0.0 alpha was released on February 10, 2020.

So, what did happen during these two years, totalling 1920 commits of work, since version 2.5 ? Let’s take a look at it !

A metric ton of user interface work

An image speaks a thousand words:

August 28, 2017 to August 17, 2021 😊 thanks to all the fine people involved ! @ossia_io #intermedia #ossia #ossiascore #musicsoftware #newmediaart #floss #opensource #freesoftware pic.x.com/ROQocR1ShD
— jm (@jcelerie) August 17, 2021

Many thanks in particular to Akané who spent a lot of time improving colours, sizes, positioning, etc. to make everything much more harmonious, pretty, and enjoyable to use.

Proper documentation

Julien and Thibaud contributed a neat documentation for version 3, which was sorely missing in the prior versions.

In particular, a lot of techniques are described in the first two chapters, which act like a tutorial of sorts:

Node-based view

Intervals can now be viewed in nodal mode to easily interconnect their processes, with a switch in the transport bar. Purely non-temporal processes (VSTs, etc.) will always show up as a small nodal slot in intervals.

Video pipeline

We developed a complete GPU-based video processing pipeline, which allows to generate, transform, filter, and modify visuals without affecting CPU usage.

FFMPEG is used for decoding videos, which means that most formats should be supported. Additionally, the HAP codec is independently supported, to ensure that we get the most out of the file format and decode it entirely on GPU.
We support applying shader-based filters on these videos and editing these shaders live, during the execution of a score. Interactive Shader Format, the well-known Vidvox technology is used as the main shader system.
The render graph is based on Qt RHI: it is possible to leverage Vulkan, OpenGL, Metal, or Direct3D for rendering. For version 3, only OpenGL will be officially supported, but it is easy to try the other APIs in the application settings.
Standard camera input is supported ; proper Spout and Syphon support are targeted for a 3.x release. NDI input and output is being developed in a separate add-on which will be shipped as part of the package manager.

Musical metrics and quantization

Support for tempo, musical bars, and quantization has been introduced, as well as a snap-on-grid system.

Time-streching is supported through different algorithms (resampling with libsamplerate, pitch-preserving timestretch with RubberBand).
It is now possible to sync the triggering of a trigger, to the beginning of the following bar for instance.
Poly-rhythm and poly-tempo are supported: each interval can have its own tempo and metrics timeline which applies recursively to its children. For instance, the parent scenario can be in a 4/4 metric, while a child scenario can be at 3/4.
It is also possible to play and stop any interval during execution in a quantized way, to enable live-looping workflows.

Presets

A preset mechanism has been introduced for all processes, to allow to save and restore processes’s state easily and to allow to create a great user library of presets.

If you make cool presets, you are more than welcome to contribute them to the official repository !

Graph intervals

This is the main breaking change in ossia score 3: in version 1 and 2, looping was handled by a specific process. In version 3, we redid that in a much more general way, by allowing to create a new type of interval which can cycle back to any other part of the score.

This means that it is now possible to create state-machine-like scores, where one part plays after the other, which then goes back to the first part, which is particularly useful for installations, and more generally very arbitrary changes and loopings in a score, we look forward to what you are going to create with it !

Package manager

In the settings, the “Package” tab allows to download add-ons and media packs.

Pure Data integration

Thanks to the libpd project, we are able to provide a fairly complete integration of Pure Data patches as score processes. Read more on the documentation !

C++ JIT support

JIT compilation support allows to live-code C++ extensions directly from within the software, just like it was possible with the Javascript process, but with greatly improved performance (and stronger real-time guarantees).

This will be an experimental feature for the 3.0 release cycle.

Along with that add-on, comes the upcoming stable plug-in API, currently being worked on here. The end goal is to ship plug-ins as source form in the package manager: this will allow them to be compiled knowing the user’s host CPU’s capabilities, thus giving peak performance by leveraging SSE, AVX, etc. whenever possible.

For instance, Maria Paula implemented the audio oracle and VMO improvisation algorithms in C++ as score plug-ins, which are then going to be integrated through the package manager:

Théo Barbé has been working on a video mapping plug-in, and Arthur Fourcadet on a GPU particles system ; both will also be released as add-ons.

Remote control

The remote control API, introduced at the end of 2.5.2, has been leveraged by a team of students to create a neat app for tablets, which can be used to control parts of the score from a mobile device very easily. Our intern Antoine has been improving it over the summer in order to make it useable from web browsers directly. More on this in a future blog post !

Smaller improvements

We added a built-in looping feature for individual processes, to easily loop audio files by simply extending them, like in most audio sequencers.
Many small and useful processes have been introduced, go discover them in the docs !
Intervals can act as mixing busses, available in a panel at the bottom.
Integration of the OLA DMX fixture library, to allow to automate lighting hardware more easily.
Official support for running score on Raspberry Pi 3 and 4, with an official release build. This build is able to use EGLFS to show video content directly in the most performant way for art installations.

Roadmap to v4

The main work for v4 will concentrate on a few major things: a stable plug-in API, MIDI 2.0, user interface improvements for the authoring and scripting, and recording. In addition, as usual, a lot of small-ish new features and improvements may make their way into the software :-)

In particular, most of the project management has moved to Github Projects: the tools is far from perfect but allows to very easily link project steps to issues and centralizes things very efficiently.

Proper Unreal Engine integration

The work done since 2019 has been achieved thanks to a generous MegaGrant from Epic, allowing Jean-Michaël to work full-time on improving score in order to provide an integration with Unreal Engine.

Part of that work has gone into making a prototype of libossia integration in UE, which did not work and required a refactor of all the network internals of libossia, from a threaded model where every protocol was living its life freely in one or multiple threads, to a much simple event-based model leveraging Boost.Asio, which will more easily integrate into the Unreal event loop.

This refactor is pretty much finished and will now enable work on making a library version of score which will then be embeddable in UE and other creative coding environments ; what remains to be specified technically is the efficient passing of sound and video data to other environments, especially when GPU processing is involved.

Stable plug-in API

The v3 to v4 release cycle will be about stabilizing things: we aim to finish the work on the execution engine to make sure that it is possible to write score plug-ins without any compromise in performance, in order to create a lasting and vibrant ecosystem of extensions to score.

As such, a lot of work is going into defining the ideal plug-in API ; the previous blog post gives hints at the path we are taking for that API.

Support for MIDI 2.0

Multiple ossia.io members are part of the Intermedia Mapping and Scripting MIDI Special Interest Group. As part of the SIG, our mission is to find ways to make sure that MIDI 2.0 is a first-class protocol for all intermedia needs ; the main way to do this will be to first try to implement MIDI 2.0 in libossia and score, and see what is missing or problematic to give recommendations to the MIDI SIG.

Timeline UX improvements

This concerns all the improvements to the central score view, in order to make it as easy as possible to author temporal content.

Things that are being evaluated for the version 4 are for instance:

Better support for sequences of automations and controls on the same parameters.
Track-like processes for sound and MIDI data, to allow applying effects more easily to a sequence of sounds.
Support for saving and reloading presets of processes as part of the timeline.
Parametric processes and variables: how to have a “configurable” process preset. That would enable for instance having a scenario preset for controlling a parameter with multiple automations and states, and being able to change the address being controlled by that preset in a single place in the user interface.
Hierarchical data support for types other than sound: for instance, automatically mixing video content in the parent like it is being done for sound right now.

The actual ideas are being put in our project board, everyone is free to participate by requesting features on the forum or on the chat :-)

Proper recording mechanism

Right now we only support a mechanism for recording data-like addresses from the device explorer to the score: OSC messages, MIDI controls, etc.

The goal for the version 4 is to work on a unified system for recording all kinds of media: sound, MIDI notes, video. This needs a better implementation than what we have now - the current recording facilities were implemented back in 2015 for the sake of a deadline and have not been touched since. In particular, they are not integrated at all with the execution data flow.

Longer term roadmap

The long-term vision for ossia score has one definite use-case: it should ultimately be possible to:

Open a web browser to our web version.
Join a collaborative scoring session, edit sound, video, etc.
Press “play”, and have the score’s execution on every browser synchronize the way it was specified by the composers: for instance, a first interval plays on user 1’s browser, and a second interval plays on user 2’s browser ; user 1 receives the sound that is generated by that interval, processes it, and sends it back to user 2.
Press “export”, and get a bundled webassembly player, which contains a compiled and optimized version of the score, either for instance as a native Windows / Mac / Linux executable, or as a WASM bundle that can then be embedded in another web page.

We are roughly 15% towards that vision, which requires a lot of different parts working together. Most importantly, the transfer of data and media assets between machines should be as efficient as possible, and has not been done at all until now.

WebAssembly port

Right now, one can make very simple scores in our web prototype, but this is not useable in any production facility. We would like to be able to export scores using sound, video, Faust plug-ins, … as a WASM object which could then be embedded in a web page.

Collaborative edition and distributed execution

These capabilities have been prototyped but not kept up to date with the rest of the software. A recent contract work has helped making the codebase compile again, it remains to fix the synchronisation issues between computers.

Contributing to ossia score

ossia is a free software project at its core, and the free software values we hold dearly.

Thanks for reading, and have fun with the beta !

Achieving generic bliss with reflection in modern C++

Thu, 22 Jul 2021 00:00:00 +0000

Reflection is often presented as a feature that makes software harder to understand. In this article, I will present ways to approximate some level of static reflection in pure C++, thanks to C++17 and C++20 features, show how that tool can considerably simplify a class of programs and libraries, and more generally enable ontologies to be specified and implemented in code.

What we aim to solve

Imagine that you are writing a neat algorithm. You’ve worked on it for a few years ; it produces great results and is now ready to be shared to the world. Let’s say that this algorithm is a noise generator.

You’d like that noise generator to easily work in a breadth of environments: in 2D bitmap manipulation programs (Krita, GIMP, …), in audio and multimedia software (PureData, Audacity, ossia score, Mixxx…), 3D voxel editors, etc.

Your algorithm’s implementation more-or-less looks like this:

// [[pre: alpha >= -1. && alpha <= 1. ]]
// [[pre: 11 <= beta && beta < 247 ]]
// [[post ret: ret >= 0. && ret <= 1.]]
auto noise(
    float input,
    float alpha,
    int beta)
{
  // set of extremely complex operations involving the inputs to the algorithm
  return std::fmod(std::pow(input, alpha), float(beta)) / beta;
}

You are proud of your neat results, prepare conference papers, etc… but ! Now is the time to implement your noise algorithm in a set of software in order to have it used widely and become the next industry standard in procedural noise generation.

If you are used to working in C#, Java, Python, or any other language more recent than 1983, the solution may at this point seem trivial. Sadly, in C++, this has been unordinately hard to implement until now, especially when one aims for as close as possible to a zero-runtime cost-abstraction.

On the other hand, if you implement your algorithm in C#, Java, or Python, having it useable from any other runtime environment is a massive challenge, as two VMs, often with their own garbage collection mechanism, etc… now have to cooperate. Thus, for something really universal, a language than can compile to native binaries, with minimal dependencies, is the easiest way to have a wide reach. In particular, most multimedia host environments are written in a native language and expect plug-ins conforming to operating system DLLs, executable and dynamic linker APIs: ELF, PE, Macho-O ; dlopen and friends. There aren’t that many suitable candidates with a high enough capacity for abstraction: C++, Rust, D without GC, Zig. Since most of the media host provide C or C++ APIs, and C does not have any interesting form of reflection, C++ is the natural, minimal-friction choice.

This post is a hint of how much better and easier life is with true reflection as available in other languages, and in particular how attribute reflection and user-defined attributes, would make one’s life. And most importantly, what kind of abstracting power “reflective programming” holds over existing generic programming techniques in C++: macro-based metaprogramming, template-based metaprogramming (with e.g. CRTP being commonly used for that).

The problem domain

The software in which we want to embed our algorithm should be able to display UI widgets adapted for the control of alpha and beta, whose bounds you have so painstakingly and thoroughly defined. Likewise, the UI widgets should adapt to the type of the parameter ; a spinbox may make more sense for beta, and a slider, knob, or any kind of continuous control for alpha.

Maybe you’d also like to serve your algorithm over the network, or through an IPC protocol like D-Bus. Again, you’d have to specify the data format being used.

If for instance you were using the OSC protocol, to make your algorithm controllable over the network, messages may look like:

/noise/input ,f 0.123
/noise/alpha ,f 13.5
/noise/beta ,i 17

Maybe you’d also like to serialize your algorithm’s inputs, in order to have a preset system, or just to exchange with another runtime system expecting a serialized version of your data. In JSON ? YAML ? Binary ? Network-byte-order binary ? GLSL std140 ? So many possibilities !

Hell on earth

For every protocol, host environment, plug-in system, … that you want to provide your algorithm to, you will have to write some amount of binding code, also often called glue code.

How does that binding code may look, you ask ?

Let’s look at some examples from around the world:

Making a fade algorithm in PureData : a class is constructed at run-time, with custom t_object, t_float, t_inlet etc… types, some of which requiring calls to various runtime allocating functions. Lots of not-very-safe-looking casts (but it’s C, there’s not a lot of choice).
Noise generator for Max/MSP’s Jitter, using OpenCV. Same as PureData, with macros sprinkled on top. Wanna get a floating-point value input by the user ? Lo and behold

void cv_jit_noise_set_stddev(t_cv_jit_noise *x, t_symbol *s, short argc, t_atom *argv)
{
	if (x && argc > 0) {
		for (short i = 0; i < x->m_dims; i++) {
			short j = i < argc ? i : argc - 1;
			x->m_stddev[i] = atom_getfloat(argv + j);
            ...
        }
    }
}

What happens if argv + j isn’t a float but a string ? Let’s leave that for future generations to discover !

Audio filter suitable for use as a VST. Notice how the parameters to the algorithms are handled in switch/case 0,1,2... ; thankfully this is all generated code from the Faust programming language. What happens if at some point a parameter is removed ? Better have good unit tests to catch all the implicit uses of each parameter…
OpenFX image filter: here’s how one says that the algorithm has a bounded input widget (e.g. a slider going from 0 to 10 with a default value of 1):

gPropertySuite->propSetString(paramProps, kOfxParamPropDoubleType, 0, kOfxParamDoubleTypeScale);
gPropertySuite->propSetDouble(paramProps, kOfxParamPropDefault, 0, 1.0);
gPropertySuite->propSetDouble(paramProps, kOfxParamPropDisplayMin, 0, 0.0);
gPropertySuite->propSetDouble(paramProps, kOfxParamPropDisplayMax, 0, 10.0);

Hopefully you don’t forget all the incantations’s updates when you decide that this control would indeed be better as an integer !

A very good example is the 1€ filter input filtering algorithm. The actual algorithm can fit in a few dozen lines. The Unreal Engine binding is almost 600 lines !
Things like iPlug are a bit more sane, but we still have to triplicate our parameter creation / access: in an enum in the hpp, in the constructor and finally in ProcessBlock where we get the actual value. This is still a whole lot of work versus JUST ACCESSING A FLOAT IN A STRUCT !!11!1!!
A Krita plug-in for noise generation – here Qt’s QObject run-time property system is used to declare and use the algorithm controls. That also means inheriting from Qt’s QObject, which has a non-negligible memory cost.
Wanna receive messages through OSC ? Make the exceptions rain !
Wanna expose your algorithm to another language, such as Python ? Get ready for some py::<>’y boilerplate.

As such, one can see that:

There is no current generic way for writing an audio processor in PureData, and have it work in, say, Audacity, Ardour or LMMS as a VST plug-in, expose it through the network… Writing a PureData external ties you to PureData, and so does writing a Krita plug-in. It’s the well-known “quadratic glue” problem: there are N algorithms and M “host systems”, thus NxM glue code to write.
All the approaches are riddled with unsafety, since the run-time environments force the inputs & outputs to the algorithm to be declared dynamically ; thus, if you make an error in your call sequence, you rely on the runtime system you are using to notice this error and notify you (e.g; if you are lucky you’ll get an error message on stdout ; but most likely a crash).
All the approaches require duplicating the actual parameters of your algorithm, e;g. our alpha, beta, once as actual C++ variables, once as facades to the runtime object system you are interacting with.

Of course, the above list is not an indictment on the code quality of those various projects: they simply all do as well as they can considering the limitations of the language at the time they were originally written, in some cases multiple decades ago.

We will show how reflection allows to improve on that, and in particular get down to N+M pieces of code to write instead of NxM.

Problem statement

Basically: there’s a ton of environments (also called “hosts”, “host APIs” in the remainder of this article) which define ad-hoc protocols or object systems. Can we find a way to make a C++ definition (“algorithm”, “plug-in” in the article) which:

Does not depend on any pre-existing code: doesn’t inherit from a class, doesn’t call arbitrary run-time functions, etc. The definition of the algorithm shall be writable without having to include anything, even standard headers (discounting of course whatever third-party library is required for the algorithm implementation).
Does not use anything other than structures of trivial, standard-layout types. No tuples, no templates, no magic, just structs containing float, int and not much more. This is because we want to be able to give the simplest possible expression of a problem. C++ is often sold as a language which aims to leave no room for a lower-level language. The technique in this post is about leaving no room for a simpler and more readable implementation of an algorithm, while maintaining the ability to control its inputs and outputs at run-time. Ideally, that would lead to a collection of such algorithms not depending on any framework, except optional concept definitions for a given problem domain. Of course, once this works, a specific community could choose to define its core concepts and ontologies through a set of standard-library-like-types, e.g. string_view, array or span-like types.
Does not duplicate parameter creation: defining a parameter should be as simple as adding a member to a structure. The parameter’s value should not be of a complicated, custom library type; just using int or float should work. At no point one should have to write the name of a variable twice, e.g. with a macro system such as Boost.Fusion with BOOST_FUSION_ADAPT_STRUCT, with Qt / copperspice / verdigris property systems, or with pybind-like templates: remember, we do not want our algorithm code to have any dependency !
Allows to specify metadata on parameters: as one could see, it is necessary to be able to define bounds, textual descriptions, etc… for the inputs to the algorithm. For instance, the algorithm author may want to define a help text for each of the parameters, describing how each control will affect the result.
Allows that definition to be automatically used to generate binding code to any of the environments, protocols, runtime systems mentioned above, with for only tool a C++20 compiler.

Massaging the problem

Sadly, C++ does not offer true reflection on any entity: from the generic function noise defined above, it would be fairly hard to extract its parameter list, and reconstruct what we need to perform the above. Likewise, due to the lack of user-defined attributes, one wouldn’t be able to tag the input / output parameters, to give them a name, bounds, etc.

We will however show that with very simple transformations, we can reach our goals !

First transformation: function to class

This transformation is commonplace in C++: classes / structs are in general more convenient to use than straight function pointers. They are easier to pass as arguments, work better with the type system as template arguments, etc.

Let’s apply it:

// noise.hpp
#pragma once
struct noise
{
  float alpha;
  int beta;

  /* constexpr_in_some_future_standard */
  float operator()(float input) const
  {
    return std::fmod(std::pow(input, alpha), float(beta)) / beta;
  }
};

Thankfully, the actual implementation does not change ; we merely put some arguments as struct members instead. If the algorithm is complex with many settings and toggles, it is likely that this was already the case in your implementation.

What if, dear reader, I told you that, as of C++17 (and actually even 14 if using a non-legal hack), this is pretty much enough for achieving three of our four goals (with some limitations, mainly on the number of members) ?

Mapping our class to a run-time API automagically

Assume the following imaginary run-time API for doing some level of processing, in cross-platform C89:

typedef void* lib_type_t;

enum lib_argument_types {
    kFloat, kInt
};

lib_type_t lib_define_type(const char* name);
void lib_add_float(lib_type_t handle, const char* name, float* ptr, float min, float max);
void lib_add_int(lib_type_t handle, const char* name, int *ptr, int min, int max);

// Vararg is a list of lib_argument_types members, defining the arguments of the function.
void lib_add_method(lib_type_t handle, const char* name, void* func, void* context, lib_argument_types input, lib_argument_types output); //some APIs may put varargs here instead

To register our process to thar imaginary API, one may write the following, which would then be compiled as a .dll / .so / .dylib and be loaded by our runtime system through dlopen and friends:

noise algo;
void process(void* context, float* out, const float* in)
{
  noise& algo = *static_cast<noise*>(context);
  *out = algo(*in);
}

lib_type_t main()
{
  auto r = lib_define_type("noise");
  lib_add_float(r, "alpha", &algo.alpha, 0., 1.); // oops
  lib_add_int(r, "beta", &algo.beta, 11, 247);
  lib_add_method(r, "process", reinterpret_cast<void*>(process), &algo, kFloat, kFloat);
  return r;
}

What we want, is simply to use C++ to generate all the code above automatically. That means, most importantly, to call the relevant lib_add_* function for each parameter with the correct arguments.

Enumerating members

This is trivial, thanks to a library, Boost.PFR, which technically works from C++14 and up. Note that the library is under the Boost umbrella but does not have any dependencies and can be used stand-alone. The technique is basically a band-aid until we get true reflection: it counts the fields by checking whether the type T is constructible by N arguments of a magic type convertible to anything, and then uses destructuring to generate tuples of the matching count.

In a nutshell:

auto tie_as_tuple(auto&& t, size<1>) {
  auto&& [_1] = t;
  return std::tie(_1);
}
auto tie_as_tuple(auto&& t, size<2>) {
  auto&& [_1, _2] = t;
  return std::tie(_1, _2);
}
// etc... computer-generated

It opens a wealth of possibilities: iterating on every member, performing operations on them, etc. ; the only restriction being: the type must be an aggregate. Thankfully, that is not a very hard restriction to follow, especially if we want to write declarative code, which lends itself pretty well to using aggregates.

Let’s for instance write a function that takes our struct and generates the lib_add_float / lib_add_int calls:

struct bind_to_lib {
  lib_type_t handle;

  void register_parameters(auto& algo)
  {
    struct {
        bind_to_lib& self;
        void operator()(float& f) const noexcept {
          lib_add_float(self.handle, "???", &f, ???, ???);
        }
        void operator()(int& i) const noexcept {
          lib_add_int(self.handle, "???", &i, ???, ???);
        }
    } visitor;
    boost::pfr::for_each_field(algo, visitor);
  }
};

This gets us 90% there: if our C API was just lib_add_float(lib_type_t, float*); that blog stop would stop right there !

But, as it stands, our API also expects some additional metadata: a pretty name to show to the user, mins and maxs…

Second transformation: ad-hoc types for parameters

This transformation is mechanical, but complexifies our code a little bit. We will change each of our parameters, into an anonymous structure containing the parameter:

float alpha;

becomes

struct {
  float value;
} alpha;

And at this point, it becomes easy to add metadata that will not have a per-instance cost, unlike a lot of runtime systems (for instance QObject properties used in Krita plug-ins):

struct {
  constexpr auto name() { return "α"sv; }
  float value;
} alpha;

The code sadly uglifies a little bit:

struct noise
{
  struct {
    constexpr auto name() { return "α"; }
    constexpr auto min() { return -1.f; }
    constexpr auto max() { return 1.f; }
    float value;
  } alpha;
  struct {
    constexpr auto name() { return "β"; }
    constexpr auto min() { return 11; }
    constexpr auto max() { return 247; }
    int value;
  } beta;

  float operator()(float input) const
  {
    return std::fmod(std::pow(input, alpha.value), float(beta.value)) / beta.value;
  }
};

There isn’t a lot of wiggle room to improve. It is not possible to have static member variables in anonymous structs ; if one is willing to duplicate the name of the struct, it’s possible to get things down to:

struct alpha {
  static constexpr auto name = "α";
  static constexpr auto min  = -1.f;
  static constexpr auto max  =  1.f;
  float value;
} alpha;

How user-defined attributes and attribute reflection would help

Now, if we were in, say, C#, what we’d most likely write instead would instead just be:

[Name("α")]
[Range(min = -1.f, max = 1.f)]
float alpha;

Simpler, isn’t it ? How neat would it be if we had the same thing in C++ ! There is some work towards that in Clang and the lock3/meta metaclasses clang fork.

We could even try (okay, that’s a little bit far-fetched) to read the pre/post conditions from C++ contract specification when it finally lands, in order to have to specify it only once !

Although in practice some methods may be needed: for instance, multiple APIs require the user to provide a method which will from an input value, render a string to show to the user.

Updating our binding code

We now have to go back and work on the binding function implementation: the main issue is that where we were using the actual type of the values, boost::pfr::for_each_field will give us references to anonymous types (or, even if not anonymous, types that we shouldn’t have knowledge of in our binding code).

In our case, we assume (as part of our ontology), that parameters have a value. This is a compile-time protocol.

Thankfully, a C++20 feature, concepts, makes encoding compile-time protocols in code fairly easy. Consider a member of our earlier visitor:

void operator()(???& f) const noexcept
{
  lib_add_float(r, ???, ???, ???, ???);
}

We can for instance fill it that way :

// we are writing the binding code, here everything is allowed !
#include <concepts>
...
void operator()(auto& f) const noexcept
  requires std::same_as<decltype(f.value), float>
{
  lib_add_float(r, f.name(), &f.value, f.min(), f.max());
}

And that would work with our current noise implementation. But what if the program author forgets to implement the name() method ? Mainly a not-so-terrible compile error:

<source>:30:23: error: no member named 'name' in 'noise::(anonymous struct at <source>:18:5)'
  lib_add_float(r, f.name(), &f.value, f.min(), f.max());
                    ~ ^
<source>:48:12: note: in instantiation of function template specialization '(anonymous struct)::operator()<noise::(anonymous struct at <source>:18:5)>' requested here
    visitor(n.alpha);
           ^

If our API absolutely requires a name(), and a value, concepts are very helpful:

template<typename T, typename Value_T>
concept parameter = requires (T t) {
  { t.value } -> std::same_as<Value_T>;
  { t.min() } -> std::same_as<Value_T>;
  { t.max() } -> std::same_as<Value_T>;
  { t.name() } -> std::same_as<const char*>;
};

Our code becomes:

void operator()(parameter<float> auto& f) const noexcept
{
  lib_add_float(r, f.name(), &f.value, f.min(), f.max());
}

Forgetting to implement name() now results in:

<source>:44:5: error: no matching function for call to object of type 'struct (anonymous struct at <source>:34:5)'
    visitor(n.alpha);
    ^~~~~~~
<source>:35:14: note: candidate template ignored: constraints not satisfied [with f:auto = noise::(anonymous struct at <source>:18:5)]
        void operator()(parameter<float> auto& f) const noexcept
             ^
<source>:35:25: note: because 'parameter<noise::(anonymous struct at <source>:18:5), float>' evaluated to false
        void operator()(parameter<float> auto& f) const noexcept
                        ^
<source>:29:7: note: because 't.min()' would be invalid: no member named 'min' in 'noise::(anonymous struct at <source>:18:5)'
  { t.min() } -> std::same_as<Value_T>;
      ^

Whether that constitutes an improvement in readability of errors in our specific case is left as an exercise to the reader.

But, what if our algorithm doesn’t actually need bounds ? We’d still want it to work in a bounded host system, right ? The host system would just choose arbitrary bounds that make sense for e.g. an input widget.

In this case, we’d get a combinatorial explosion of concepts: we’d need an overload for a parameter with a name and no range, an overload for a parameter with a range and no name, etc.

Handling optionality

As an algorithm author, you cannot specify every possible metadata known to man. We want our algorithm to be future-proof: even if refinements can be added, we want the code we write today to still be able to integrate into tomorrow’s host.

Thankfully, the age-old notion of condition can help here ; in particular compile-time conditions depending on the existence of a member.

C++20 makes that trivial:

void operator()(auto& f) const noexcept
  // We still need our "requires" here, or a simpler concept
  // in order to have the right overload be selected.
  requires std::same_as<decltype(f.value), float>
{
  const char* name = "Parameter";
  float min = 0.f, max = 1.f;
  if constexpr(requires { name = f.name(); })
    name = f.name();
  if constexpr(requires { min = f.min(); })
    min = f.min();
  if constexpr(requires { max = f.max(); })
    max = f.max();

  lib_add_float(r, name, &f.value, min, max);
}

This way, the algorithm has maximal flexibility: it can provide the bare minimal metadata for a proof-of-concept, or give as much information as possible.

This last part works in Clang and GCC, but MSVC’s concepts implementation does not support it yet.

Calling our code

There’s not much difference with the previous technique when we want to call our process (operator()) function.

What we cannot do without reflection & code generation (metaclasses) is an entirely generic transformation from one of our algorithm’s processing method, which, depending on the problem domain, could have any number of inputs / outputs of various types, to arbitrary run-time data. For instance, audio processors generally have inputs and outputs in the form of an array of channels of float / double values, plus the amount of data to be processed:

void canonical_audio_processor(float** inputs, float** outputs, int frames_to_process);

While image processors would instead look like:

void canonical_image_processor(unsigned char* data, int width, int height);

There’s no practical way to enumerate all the possible sets of arguments.

Thus, the author of the binding code has the responsibility of adapting the expected ontology for algorithms to the API we are binding to.

struct bind_to_lib {
  lib_type_t handle;

  template<typename T>
  void register_process(T& algo)
  {
    auto process = [] (void* context, float* out, const float* in) {
      T& algo = *static_cast<T*>(context);
      *out = algo(*in);
    };
    lib_add_method(handle, "process", reinterpret_cast<void*>(process), &algo, kFloat, kFloat);
  }
}

Nothing prevents multiple cases to be handled: for instance, some plug-ins may have a more efficient, array-based, implementation for their process ; some hosts may be able to use that if available:

template<typename T>
void register_process(T& algo)
{
  if constexpr(std::invocable<T, const float*, float*, std::size_t>)
  {
    auto process = [] (void* context, float* out, const float* in, std::size_t n) {
      // ...
      algo(in, out, n);
    };
    lib_add_method(handle, "process_array", reinterpret_cast<void*>(process), &algo, kFloat, kFloat);
  }
  else if constexpr(std::invocable<T, float, float>)
  {
    auto process = [&] (void* context, float* out, const float* in) {
      // ...
      *out = algo(*in);
    };
    lib_add_method(handle, "process", reinterpret_cast<void*>(process), &algo, kFloat, kFloat);
  }
}

An host which only supports array-based computations would instead write:

template<typename T>
void register_process(T& algo)
{
  if constexpr(std::invocable<T, const float*, float*, std::size_t>)
  {
    auto process = [&] (float* out, const float* in, std::size_t n) {
      // ...
      algo(in, out, n);
    };
    lib_add_method(handle, "process", reinterpret_cast<void*>(process), &algo, kFloat, kFloat);
  }
  else if constexpr(std::invocable<T, float, float>)
  {
    auto process = [&] (float* out, const float* in, std::size_t n) {
      // ...
      for(std::size_t i = 0U; i < n; i++)
        out[i] = algo(in[i]);
    };
    lib_add_method(handle, "process", reinterpret_cast<void*>(process), &algo, kFloat, kFloat);
  }
}

Going thread-safe

Suppose that our C host API specifies that the process method is run in a separate thread, for efficiency concerns. Such an API’s lib_add_float function could look like this:

void lib_add_float(
  lib_type_t handle,
  const char* name,
  float (*getter)(void*),
  void (*setter)(void*, float),
  void* context
);

where context would be an object passed to getter and setter so that the actual float could be found.

getter and setter could be called from any of the host’s threads, e.g. the main or UI thread for instance, while process would be called from a separate thread specifically.

Thus, our actual float needs some protection. Now, our program has the added requirement of not using locks: the algorithm could be used from a real-time system.

What we can do is, transform our list of parameters into atomic<T> types, at compile-time. A simple way for this is through any of the common C++ type-based metaprogramming libraries, which are able to transform tuples: in our case we’ll use Boost.MP11 ; other alternatives are Brigand, Metal, etc.

template<typename T>
struct binding_to_lib {
  // parameters_type will look like tuple<float, int>
  using parameters_type = decltype(boost::pfr::structure_to_tuple(std::declval<const T&>()));

  // parameters_type will look like tuple<std::atomic<float>, std::atomic<int>>
  using atomic_type = mp_transform<std::atomic, parameters_type>;

  // our algorithm's aggregate struct, we're allocating it as part of the binding for more simplicity
  T implementation;

  atomic_type parameters;

  binding_to_lib() { /* register everything with the host API */ }
};

From this, our binding methods would be changed to look like:

void register_process()
{
  auto process = [] (void* context, float* in, float* out) {
    // Get our objects back from the context pointer
    auto& self = *static_cast<decltype(this)>(context);
    T& algo = self.implementation;

    // Load all the atomics into the "algo" object
    load_all_atomics(self.parameters, boost::pfr::structure_tie(algo));

    // Apply the algo
    *out = algo(*in);
  };

  lib_add_method(handle, "process", reinterpret_cast<void*>(process), this, kFloat, kFloat);
}

void load_all_atomics(const auto& atomics, auto& actual)
{
  [&] <std::size_t... Index> (std::integer_sequence<std::size_t, Index...>) {
    using namespace boost::pfr;
    ((get<Index>(actual) = get<Index>(atomics).load(std::memory_order_relaxed)), ...);
  }(std::make_index_sequence<std::tuple_size_v<parameters_type>>());
}

The load_all_atomics function is a bit dense to read, here’s a spaced-out and simplified version:

void load_all_atomics(const auto& atomics, auto& actual)
{
  // Template lambda (C++20 feature): one used to have to write a separate
  // load_all_atomics_helper function as there was no way to retrieve the parameter pack,
  // which pollutes the namespace
  auto helper = [&] <std::size_t... Index> (std::integer_sequence<std::size_t, Index...>)
  {
    using namespace boost::pfr; // for `get`

    // Fold expression which goes through every member of the tuples - which have the same size.
    // If we don't choose a memory order for loading, it defaults to the safest but also least efficient sequentially-consistent order, which does not matter here: our parameters are all entirely independent, we just care about the atomic access to the variable itself (famous last words).
    ((get<Index>(actual) = get<Index>(atomics)), ...);
    // ^ equivalent to
    // get<0>(actual) = get<0>(atomics);
    // get<1>(actual) = get<1>(atomics);
    // get<2>(actual) = get<2>(atomics);
  };

  // std::integer_sequence<std::size_t, 1, 2>
  auto sequence = std::make_index_sequence<std::tuple_size_v<parameters_type>>();

  // Call our lambda
  helper(sequence);
}

Note that in the end, compilers will happily inline all that into a couple of mov instructions :-)

Its conjugate function, store_all_atomics is left as an exercise to the reader.

Another interesting function that one can write is the function that will perform an operation on the nth parameter, n being known only at run-time, as some APIs are index-based instead of pointer-based: parameters are identified through an index.

Here’s a solution I found, which Clang is able to optimize pretty well through what looks like a loop recombination optimization, but other compilers sadly don’t manage to:

template <typename Parameters, typename F>
void for_nth_parameter(Parameters& params, int n, F&& func) noexcept
{
  [&]<std::size_t... Index>(std::integer_sequence<std::size_t, Index...>)
  {
    ((void)(Index == n && (func(boost::pfr::get<Index>(params)), true)), ...);
  }
  (std::make_index_sequence<boost::pfr::tuple_size_v<Parameters>>());
}

Conclusion

From this, it is obvious that writing, for instance, a generic serializer, hash function, etc… that will work on any such types is trivial ; Boost.PFR already provides some amount of it. A fun exercise left to the reader would be memoization of plug-in state, for the sake of undo-redo.

Note that the algorithm could also easily be generic ; for instance, some audio plug-in APIs support working with either single- or double- precision floating-point ; one could just provide a noise<std::floating_point> algorithm if it fits the algorithm’s spec. Otherwise, the binding library would simply perform the conversion from / to the correct floating-point type if that is a meaningful thing to do.

This concept has been prototyped in, first, an API for writing plug-ins for ossia score, and then in a tentative for writing audio plug-ins, vintage.

There is one last drop of manual binding code to write: the code that ties the algorithm to the API.

// main.cpp
#include <noise.hpp>
#include <bind_to_lib.hpp>
extern "C"
DLL_EXPORT_MACRO
void plugin_main_function() { bind_to_lib<noise>(); }

There is no easy way without full static reflection to bypass that drop of code: we have to reference the name of our algorithm in the same line than our binding code at least once ; full reflection would allow to enumerate the available types instead and skip that part. There are two band-aid solutions:

State that the class containing the algorithm must have a specific name, e.g. Plugin. This does not really scale if for instance a software would like to build and bundle multiple such plug-ins statically, due to ODR ; it can be made to work with shared objects if one takes care of hiding all symbols except plugin_main_function.
Generate that code in the build system: one could easily provide a set of e.g. bind_algorithm(<name> <main_source_file>) CMake function which would generate the appropriate .cpp ; the act of porting the algorithm to a new host platform would simply be for instance forking a template repo on GitHub, and replacing the content of an src/algorithm.cpp file.

So, in the end, what we have, roughly, is:

Algorithms without dependencies on host APIs for exposing themselves to GUI software, etc.
Independent introspection of these algorithms.

What remains is, as a community, to specify the ontologies / concepts that a given algorithm can be made to fit in: for instance, for audio plug-ins, the LV2 specification has done a great deal of work towards that ; similar work could be done for graphics algorithms, serialization systems, etc.

This work could be encoded in C++ concepts, maybe with inspiration from the various Haskell typeclasses or Rust traits libraries: then, if as an algorithm author I want to make sure that my algorithm will be able to be used by audio, video, … software, I’d just clone a concept-checking library and see which concepts my code does (and does not) support ; an algorithm which takes a float and outputs a float would likely have a very wide applicability.

A further blog article will present how one can leverage this to build data-flow graphs either at compile-time or at run-time.

Summer internships

Fri, 25 Jun 2021 00:00:00 +0000

Summer internships in ossia.io

This year we are extremely lucky in having five great interns and students working on the software.

This summer, they are going to work on improving the score documentation, working on the remote control, implementing audio oracle algorithms, and implementing new fancy graphics features.

Aldwin Baïetto

My name is Aldwin Baïetto and I am 21 years old. I am a first year student at the ENSC (École Nationale Supérieur de Cognitique). I study the user experience and every aspect that is related to users. I am passionate about technology and IA. I also protect some theatre in my free time.

Aldwin helps improving the overall ergonomy and user experience of the system after a successful ENSC school project led with Isabelle Oberlin and Charlotte Marty ; he will continue their work and also improve the user documentation.

Théo Barbé

Student in computer science and engineering at ENSEIRB-MATMECA, I want to specialize in multimedia. Passionate about music, I will take off soon towards Canada for new adventures.

Théo is working on the graphics and video pipeline of ossia score. He is working on supporting advanced video mapping features in the software in order to open new use cases.

Maria Paula Carrero

My name is Maria Paula Carrero, I’m a Colombian Computer Science student. I’m a hard-working, self-starter, responsible and meticulous woman who wants to inspire women to work in STEM. I worked for the Sorbonne Paris Nord as a Research Intern and I’m working as a front-end developer in a software developing company for international clients.

I’m passionate about learning new things, exploring new languages, and programming languages to expand my knowledge. In my free time I like spending time with my family, going to the cinema, watching movies, television shows and listening to music.

Maria Paula has already provided implementations of Factor Oracle algorithms for arbitrary values as well as MIDI in ossia score, and is going to work on the audio oracle algorithm for live improvisation.

Antoine Dudouit

Through my studies in engineering school at ENSEIRB-MATMECA, I satisfy my interest in computer science, though I am also very attracted to art. So, I would like to combine the two and use computer science as an artistic tool.

Antoine is continuing the work started this year on the remote control for ossia score ; he is constantly adding new features and improving it in order to make it easy to remote control score from tablets, web browsers, etc.

Arthur Fourcadet

I am Arthur Fourcadet, 21, in second year of computer science at the engineering school, ENSEIRB-MATMECA and on my way to specialize in software engineering. Volleyball lover and really fond of music and video games, I’m always searching for new things to discover.

Arthur is working on particle systems and is going to implement various particle generation algorithms in real-time on the GPU, using compute shaders for maximum performance.

ossia | Open Software System for Interactive Applications

Case study: [re]capture

Case study: scored air filtration in [re]capture

The [re]capture project

Prototyping and producing a score

Connecting ossia to the devices

Input devices

Output devices

Test cues

Experimenting with mappings

Organizing the score

Making it loop

Score experiment: triggering a fan after a motor

Score experiment: Emergency behaviours

Mappings

Mapping a set of sensor to a module

Scripting

Global mappings

Video mapping

On squishiness and light adjustements

Mapping parameters to real-time video filters

New objects for pattern control

Calibrator

Pattern combiner

Pattern applier

Pattern sweeper

Conclusion

Generalized "value" types and prototypal operation for C++ objects

Problem position

Aggregate member name reflection

Compatible types

Value types?

Value type support in Avendish

jk: jq for any object

Primer of jq syntax

Implementation

Usage

Treating OSC nodes as objects

Example

Conclusion

Score on the Store

Getting ossia score on the Microsoft Store

Technical details

Prerequisites

Steps

Account creation

Asset setup

Package manifest setup

Important Package.appxmanifest details for Win32 apps

XML funsies

Package layout XML

Application manifest

Generating the package

Generating a certificate

Packaging

Testing the packaged app

Installing the certificate

Running the app

Checking with Windows App Cert Kit

Collaborative and distributed interactive scores

Collaborative features in score 3.0.11

Going network goblin mode

Collaborative edition features

Distributed execution

Distributing expression evaluation

Distributing interval speed

Message pit: sharing messages across the session

It even works on the WASM build

Minimum viable declarative GUI in C++

Introducing a “toolkit”

Advanced UI

Nested UI

Trivial UI backends

Conclusion

Introducing ossia score 3

Introducing ossia score 3

New features and changes since v2 and v3 beta

JSFX support

Official Raspberry Pi builds

Contributing to ossia score