Link Search Menu Expand Document

ossia

This page summarizes the systems and file formats that score can interoperate with. Availability depends on the operating system, build options, installed add-ons and external runtimes.

Operating systems

score works on Linux, macOS, Windows, and partially on the web platform.

Its development mainly happens on an ArchLinux system. As score is built with Qt, it should be portable to any system where Qt runs.

Network protocols

  • OSC (Open Sound Control): the standard intermedia protocol. It is implemented through a heavily modified version of Ross Bencina’s oscpack library.
  • OSCQuery:
  • Minuit.
  • HTTP.
  • WebSockets.
  • CoAP.
  • MQTT.
  • LSL (Lab Streaming Layer), through the optional LSL add-on.
  • Bitfocus Companion modules expose supported module controls through a device; the module’s own dependencies and configuration still apply.
  • Spatialization, provided by the SpatGRIS add-on, is a device integration for external spatial-audio software.

The optional Libmapper client is a separate integration from the scriptable Mapper device. Its current source integration has a build-interface mismatch; do not treat it as an available working protocol in an ordinary installation.

Tracking protocols records the TUIO, PSN, RTTrP, OpenTrackIO and optional OpenXR implementations. They are included in desktop builds since 3.9.0, in the Tracking category of the Add device dialog. Depth cameras stream color, depth, point clouds and IMU data from Orbbec, Kinect, Azure Kinect, RealSense and Luxonis OAK cameras.

Lighting protocols

  • Art-Net / DMX: the standard for lighting fixtures. Support is implemented through libartnet, which has been integrated inside libossia. score is able to load fixtures definitions in the open-fixture-library format.
  • LED Strip Support: Comprehensive support for LED strips including NeoPixel (WS2812), individual LED control, and strip/pane/volume layouts through Art-Net. Supports RGB, RGBW, and specialized diode configurations (Warm/Cold White, Amber, UV).
  • s.ACN / E1.31 is supported.
  • ENTTEC DMX USB Pro devices are supported (Mk1 and Mk2).
  • DMX input and output are both supported.

Hardware protocols

  • Serial port: score can read and write serial ports, including Bluetooth connections exposed by the operating system as serial ports.
  • Game pads: they are supported through the SDL2 gamepad library. Most gamepads and joysticks should work without issue.
  • Wiimotes: they are supported through the WiiUse library.
  • LeapMotion / UltraLeap: they are supported through the UltraLeap SDK (Gemini / Hyperion) which must be installed on the target computer.
  • GPIOs, ADCs, PWMs etc.: they are supported through the SimpleIO library which is a simple wrapper over the raw kernel access. This feature is only available on Linux with the relevant hardware, for instance on Raspberry Pi.
  • BLE: Bluetooth Low Energy devices are supported through the SimpleBLE. Both reading advertisments / beacons / manufacturer data and GATT services is supported.
  • GPS: score can connect to a gpsd server and expose the GPS data.
  • Phidgets has an optional Phidget22 integration in libossia. The wrapper contains a stale include path that must be corrected for a Phidgets-enabled build to work. See the reference for channel activation and input/output limitations.
  • CAN / DBC receives and decodes CAN signals through Linux SocketCAN. It is receive-only and requires a DBC database; it does not provide CAN transmission or CANopen control services.

Audio systems

  • JACK: support is implemented in libossia.
  • PulseAudio: support is implemented in libossia.
  • PipeWire: implemented in libossia and in score.
  • ALSA, the native Linux backend, supported through PortAudio. A direct implementation is also provided for instance for working with as low latency as possible on embedded devices, but it only supports output, not duplex / input.
  • CoreAudio: the native macOS backend, supported through PortAudio.
  • MME, WASAPI, WDMKS: the native Windows backends, supported through PortAudio.
  • ASIO: the low-latency pro-audio Windows backend developed by Steinberg, supported through PortAudio.
  • SDL: support is implemented in libossia. It is mainly used to provide audio for the WebAssembly build of score.

Video protocols

  • Spout is supported on Windows.
  • Syphon is supported on macOS.
  • Shmdata is supported on Linux and macOS.
  • Sh4lt provides shared-memory video input and output when built with the Sh4lt backend. See the Sh4lt device; the wider Sh4lt library’s data types are not a promise that every type is exposed by this device.
  • NDI video input and output require the NDI add-on and a loadable NDI runtime. Platform and format support depend on that runtime and build.
    • See the NDI device, including PTZ controls for compatible cameras.
  • PipeWire video provides input and output on Linux builds with that backend. It is distinct from the PipeWire audio and MIDI backends.
  • GStreamer bridges user-defined media pipelines; capabilities depend on the GStreamer runtime and installed elements.
  • FFmpeg / libav provides media-device input and output for supported files and streams. Available codecs, muxers and network protocols depend on the FFmpeg build.
  • Camera and screen capture use platform-specific capture backends.
  • GPhoto2 DSLR provides camera live-view preview and device-reported configuration when libgphoto2, its camera driver and suitable hardware are available.

Transport synchronisation

  • JACK transport: score can act as a master or a slave.
  • Timecode Synchronizer follows an incoming time position. Musical pulse generation and timestamp synchronization are different operations; choose the process appropriate to the source.
  • MIDI Sync Out sends MIDI clock, transport messages and timecode. MIDI Sync In decodes incoming synchronization into control values; it does not automatically make the whole score follow an external transport.

MIDI

All the MIDI support in score comes from the libremidi library:

For real-time communication, the following implementations are provided:

  • ALSA, through either the raw or sequencer API.
  • JACK.
  • PipeWire.
  • The native operating systems MIDI API: MME for Windows, CoreMIDI for macOS.
  • WebMIDI.

In addition, score is able to load Standard MIDI files (SMF).

Use MIDI input and MIDI output for MIDI streams. MIDI Controller additionally provides named device-map controls and a Mackie Control surface mode; these are two modes of one protocol, not separate processes.

Audio file formats

score uses FFMPEG for its audio needs.

It should support most codecs and formats listed at this page. Every standard format (WAV, W64, AIFF, MP3, OGG Vorbis, FLAC, etc.) are supported without issues.

score handles WAV files in a specific way, through the dr_wav library, to allow for memory-mapping the data for large files.

score is able to read ACID tags to devise for instance BPM info from sound files.

See the sound file process documentation for more information.

Video file formats

score uses FFmpeg to read video files. Available demuxers and codecs depend on the FFmpeg build; common choices include H.264, H.265, ProRes and transport streams (.ts).

HAP and supported DXV variants have specialized GPU texture-decoding paths. This differs from hardware video decoding: compressed texture blocks can be sampled on the GPU, while the file still needs to be read and unpacked.

The Video process provides Auto, Direct (seek) and Frame queue playback modes. Auto selects the playback path using the stream’s keyframe layout. Direct playback is useful for independently decodable frames; inter-frame codecs generally need the frame queue.

Hardware decoding

The video decoder can use FFmpeg hardware-acceleration backends, including Direct3D / DXVA on Windows, VideoToolbox on macOS, and Linux options such as VAAPI, V4L2-M2M, CUDA or QuickSync where available. Vulkan decoding is also build- and driver-dependent. Selecting a graphics renderer does not by itself guarantee hardware video decoding.

Support depends on the codec, pixel format, FFmpeg version, driver and GPU. Use the video decoding preference to select a backend; unsupported streams may still need software decoding.

Hardware rendering

score can convert many decoded pixel formats with GPU shaders rather than converting every frame to RGB on the CPU. This reduces conversion work; it does not make file decoding or transfer free.

Supported paths cover packed and planar RGB / RGBA, grayscale, YUV and YUVA; common examples include NV12, P010 / P016, planar 4:2:0 / 4:2:2 / 4:4:4, YUYV / UYVY, high-bit-depth and floating-point images, and DCI XYZ12. The precise path depends on the source format and graphics backend; other formats can require a software conversion.

Color and HDR

The video pipeline interprets source color metadata, including BT.601, BT.709, BT.2020, P3, PQ and HLG. The Video inspector’s Format and Tonemap (HDR) controls choose conversion to SDR, HDR passthrough or linear output, and the tone-mapping operator.

Decoding an HDR file is not the same as displaying HDR. The output window format, graphics backend, operating system and display must also support the requested output. For SDR displays, use tone mapping rather than assuming passthrough will produce the intended image.

See Video, Window and the graphics pipeline for the processing and output controls.

For detailed choices and limitations, see Video formats and color.

Image file formats

The Image process reads raster images through Qt’s image readers and handles SVG through a scalable SVG renderer. Common formats include PNG, JPEG and GIF; additional formats depend on installed Qt image-format support.

3D file formats

Native 3D processes are separate from QtQuick3D content embedded in a JavaScript / QML process:

  • Geometry Loader (Object Loader reference) imports geometry for mesh-oriented processing.
  • Asset Loader imports scenes from glTF / GLB and FBX, as well as OBJ, PLY, STL, OFF, SPLAT and SPZ assets. It preserves scene data where the format and importer support it; USD-family files require the optional parser add-on.
  • Scene data can be processed and rendered through the native Render Pipeline. A simple geometry path can use Model Display.
  • JavaScript / QML can host QtQuick3D scenes when the required Qt modules are installed. This is a separate rendering path, not a requirement for native glTF / FBX import.

Graphics APIs

score uses Qt RHI to target OpenGL, Vulkan, Metal, Direct3D 11 and Direct3D 12. Available backends depend on the platform and build. Advanced features such as compute shaders, storage buffers and particular texture formats additionally depend on GPU and driver support.

The shader processes cover several jobs:

See the graphics pipeline for how these processes connect.

Audio plug-ins

Desktop builds support the following audio plug-in and embedded-language systems when the corresponding backend is included. A plug-in binary must match the operating system and architecture; desktop plug-in support does not imply availability in the browser build.