Graphics rendering pipeline for ossia score. More...

Detailed Description

Graphics rendering pipeline for ossia score.

This namespace provides a rendering pipeline for GPU visuals, by the way of a rendering graph. The implementation uses the Qt RHI:

https://www.qt.io/blog/graphics-in-qt-6.0-qrhi-qt-quick-qt-quick-3d

It is mainly designed for rendering a succession of visual effects, not rendering arbitrary 3D. Shaders are written in the usual glslang -> spirv-cross fashion, with GLSL #version 450. They are then translated automatically to SPIRV, Metal, GLSL or D3D11 shaders.

The overall design is as follows :

Elements to render are represented as nodes to a graph: score::gfx::Node
Input ports are the uniforms and input textures & audio data.
Output ports generally are just the texture we are rendering to.
score::gfx::Graph contains all the nodes, and relationships between these nodes.
For each output sink (a window surface, an NDI or Spout output...), a score::gfx::RenderList is created. It contains all the nodes to render for a given output.
For each score::gfx::Node, the score::gfx::RenderList creates a score::gfx::NodeRenderer which contains the actual GPU resource handles for the sink.

Given the following graph :

[ video ] -> [ shader 1 ] -> [ screen 1 ]
                          \
                           -> [ shader 2] -> [ Spout output ]

Two render lists are created, the first being

[ video ] -> [ shader 1 ] -> [ screen 1 ]

and the second being

[ video ] -> [ shader 1 ] -> [ shader 2] -> [ Spout output ]

Rendering for these two RenderList is entirely independent - they can render at different sizes and potentially use different graphic APIs, e.g. one OpenGL and the other Vulkan. This is also done to enable threaded rendering for each of these.

We provide the most essential nodes for building interesting visual effects:

A window output node (score::gfx::ScreenNode).
Shader filters using Interactive Shader Format (https://isf.video).
Video (score::gfx::VideoNode) and image (score::gfx::ImagesNode) renderers.
The video renderer tries to do as much of the decoding as possible on the GPU: for instance YUV420, HAP, ... are decoded on the GPU. Otherwises it falls back on libavcodec to guarantee compatibility with a wide range of video formats.

Other plug-ins may provide other useful things, such as NDI I/O or particle effects.

See GfxPlugins for information on how to create your custom nodes, visual effects and video decoders.

Classes
struct	ProcessUBO
	UBO specific to individual processes / nodes. More...

struct	ModelCameraUBO
	UBO shared across all entities shown with the same camera. More...

struct	OutputUBO
	UBO shared across all entities shown on the same output. More...

struct	VideoMaterialUBO
	UBO shared across all video objects. More...

class	CustomMesh

class	GPUVideoDecoder
	Processes and renders a video frame on the GPU. More...

struct	EmptyDecoder
	Default decoder when we do not know what to render. More...

struct	HAPDecoder
	Base class for HAP ((c) Vidvox) decoding. More...

struct	HAPDefaultDecoder
	Decodes HAP basic format. More...

struct	HAPMDecoder
	Decodes HAP-M (HAP + alpha channel) More...

struct	NV12Decoder
	Decodes NV12 videos. More...

struct	PackedDecoder

struct	PlanarDecoder

struct	PackedRectDecoder

struct	RGB24Decoder

struct	YUV420Decoder
	Decodes YUV420 videos. More...

struct	YUV420P10Decoder
	Decodes YUV420 videos. More...

struct	YUV420P12Decoder
	Decodes YUV420 videos. More...

struct	YUV422Decoder
	Decodes YUV422 videos. More...

struct	YUV422P10Decoder
	Decodes YUV422 videos. More...

struct	YUV422P12Decoder
	Decodes YUV422 videos. More...

struct	YUYV422Decoder
	Decodes YUYV422 video. More...

struct	UYVY422Decoder
	Decodes UYVY422 video, mostly used for NDI. More...

struct	DepthNode

struct	geometry_input_port_vis

class	GeometryFilterNode
	Data model for geometry filters. More...

struct	GeometryFilterNodeRenderer

struct	Graph
	Represents a graph of renderers. More...

struct	ImagesNode
	A node that renders an image to screen. More...

struct	FullScreenImageNode

struct	isf_input_port_vis

class	ISFNode
	Data model for Interactive Shader Format filters. More...

struct	isf_input_size_vis

struct	MeshBuffers

struct	Mesh
	Data model for meshes. More...

struct	BasicMesh

struct	PlainMesh
	A mesh with only position attributes. More...

struct	TexturedMesh
	A mesh with positions and texture coordinates. More...

struct	PlainTriangle
	A triangle mesh with only positions. More...

struct	TexturedTriangle
	A triangle mesh with positions and texture coordinates. More...

struct	TexturedQuad
	A quad mesh with positions and texture coordinates. More...

struct	Timings
	Messages sent from the execution thread to the rendering thread. More...

struct	Message

class	Node
	Root data model for visual nodes. More...

class	ProcessNode
	Common base class for nodes that map to score processes. More...

class	NodeModel
	Common base class for most single-pass, simple nodes. More...

class	NodeRenderer
	Renderer for a given node. More...

class	GenericNodeRenderer
	Generic renderer. More...

class	OutputNodeRenderer

class	OutputNode
	Base class for sink nodes (QWindow, spout, syphon, NDI output, ...) More...

struct	RenderedPhongNode

struct	PhongNode

struct	Pass

struct	PersistSampler

struct	AudioTextureUpload

struct	RenderedISFNode

struct	SimpleRenderedISFNode

class	RenderList
	List of nodes to be rendered to an output. More...

struct	RenderState
	Global state associated to a rendering context. More...

struct	ScreenNode
	This node is used for rendering to a score::gfx::Window. More...

struct	ShaderCache
	Cache of baked QShader instances. More...

struct	TexgenNode

struct	TextNode
	A node that renders text to screen. More...

struct	image

struct	geometry

struct	Sampler
	Stores a sampler and the texture currently associated with it. More...

struct	AudioTexture
	Data model for audio data being sent to the GPU. More...

struct	Port
	Port of a score::gfx::Node. More...

struct	Edge
	Connection between two score::gfx::Port. More...

struct	Pipeline
	Useful abstraction for storing a graphics pipeline and associated resource bindings. More...

struct	TextureRenderTarget
	Useful abstraction for storing all the data related to a render target. More...

struct	Image
	Image data and metadata. More...

struct	DefaultShaderMaterial
	Utility to represent a shader material following score conventions. More...

struct	RefcountedFrame

struct	VideoFrameShare

struct	VideoFrameReader

class	VideoNodeBase

class	VideoNode
	Model for rendering a video. More...

class	CameraNode
	Model for rendering a camera feed. More...

class	VideoNodeRenderer

class	Window
	A platform window in which the content is going to be rendered. More...

Typedefs
using	Vertex = score::gfx::Node *

using	GraphImpl = boost::adjacency_list< boost::vecS, boost::vecS, boost::directedS, Vertex >

using	VertexMap = ossia::hash_map< score::gfx::Node *, GraphImpl::vertex_descriptor >

using	FunctionMessage = std::function< void(score::gfx::Node &)>

using	gfx_input = ossia::slow_variant< ossia::monostate, ossia::value, ossia::audio_vector, ossia::geometry_spec, ossia::transform3d, FunctionMessage >

using	PassMap = ossia::small_vector< std::pair< Edge *, Pipeline >, 2 >

using	PassOutput = ossia::variant< PersistSampler, TextureRenderTarget >

Enumerations
enum	ImageMode { Single , Clamped , Tiled , Mirrored }

enum	GraphicsApi { Null , OpenGL , Vulkan , D3D11 , Metal }
	Available graphics APIs to use.

enum	ScaleMode { Original , BlackBars , Fill , Stretch }
	How to resize a texture to adapt it to a viewport.

enum class	Types { Empty , Int , Float , Vec2 , Vec3 , Vec4 , Image , Audio , Camera , Geometry }

Functions
template<typename Graph_T , typename IO >
void	print_graph (Graph_T &g, IO &stream)

	Q_DECLARE_OPERATORS_FOR_FLAGS (Mesh::Flags)

void	defaultPassesInit (PassMap &passes, const std::vector< Edge * > &edges, RenderList &renderer, const Mesh &mesh, const QShader &v, const QShader &f, QRhiBuffer processUBO, QRhiBuffer matUBO, const std::vector< Sampler > &samplers, std::span< QRhiShaderResourceBinding > additionalBindings)

void	defaultRenderPass (RenderList &renderer, const Mesh &mesh, const MeshBuffers &bufs, QRhiCommandBuffer &cb, Edge &edge, PassMap &passes)

void	quadRenderPass (RenderList &renderer, const MeshBuffers &bufs, QRhiCommandBuffer &cb, Edge &edge, PassMap &passes)

TextureRenderTarget	createRenderTarget (const RenderState &state, QRhiTexture *tex, int samples)
	Create a render target from a texture.

TextureRenderTarget	createRenderTarget (const RenderState &state, QRhiTexture::Format fmt, QSize sz, int samples, QRhiTexture::Flags={})
	Create a render target from a texture format and size. More...

void	replaceBuffer (std::vector< QRhiShaderResourceBinding > &tmp, int binding, QRhiBuffer *newBuffer)

void	replaceSampler (std::vector< QRhiShaderResourceBinding > &tmp, int binding, QRhiSampler *newSampler)

void	replaceTexture (std::vector< QRhiShaderResourceBinding > &tmp, int binding, QRhiTexture *newTexture)

void	replaceBuffer (QRhiShaderResourceBindings &srb, int binding, QRhiBuffer *newBuffer)

void	replaceSampler (QRhiShaderResourceBindings &srb, int binding, QRhiSampler *newSampler)

void	replaceTexture (QRhiShaderResourceBindings &srb, int binding, QRhiTexture *newTexture)

void	replaceSampler (QRhiShaderResourceBindings &, QRhiSampler oldSampler, QRhiSampler newSampler)
	Replace a sampler.

void	replaceTexture (QRhiShaderResourceBindings &, QRhiSampler sampler, QRhiTexture newTexture)
	Replace the texture currently bound to a sampler.

void	replaceTexture (QRhiShaderResourceBindings &srb, QRhiTexture old_tex, QRhiTexture new_tex)
	Replace a texture by another in a set of bindings.

Pipeline	buildPipeline (const RenderList &renderer, const Mesh &mesh, const QShader &vertexS, const QShader &fragmentS, const TextureRenderTarget &rt, QRhiShaderResourceBindings *srb)

QRhiShaderResourceBindings *	createDefaultBindings (const RenderList &renderer, const TextureRenderTarget &rt, QRhiBuffer processUBO, QRhiBuffer materialUBO, const std::vector< Sampler > &samplers, std::span< QRhiShaderResourceBinding > additionalBindings={})
	Create bindings following the score conventions for shaders and materials.

Pipeline	buildPipeline (const RenderList &renderer, const Mesh &mesh, const QShader &vertexS, const QShader &fragmentS, const TextureRenderTarget &rt, QRhiBuffer processUBO, QRhiBuffer materialUBO, const std::vector< Sampler > &samplers, std::span< QRhiShaderResourceBinding > additionalBindings={})
	Create a render pipeline following the score conventions for shaders and materials.

std::pair< QShader, QShader >	makeShaders (const RenderState &v, QString vert, QString frag)
	Get a pair of compiled vertex / fragment shaders from GLSL 4.5 sources. More...

QShader	makeCompute (const RenderState &v, QString compt)
	Compile a compute shader. More...

QSize	resizeTextureSize (QSize img, int min, int max) noexcept
	Resize the size of a texture to fit within GPU limits.

QImage	resizeTexture (const QImage &img, int min, int max) noexcept
	Resize a texture to fit within GPU limits.

QSizeF	computeScale (score::gfx::ScaleMode mode, QSizeF viewport, QSizeF texture)
	Comput the scale to apply to a texture so that it fits in a GL viewport.

void	copyMatrix (const QMatrix4x4 &mat, float *ptr) noexcept

void	copyMatrix (const QMatrix3x3 &mat, float *ptr) noexcept

Variables
const char *	frag

Function Documentation

◆ createRenderTarget()

TextureRenderTarget score::gfx::createRenderTarget	(	const RenderState &	state,
		QRhiTexture::Format	fmt,
		QSize	sz,
		int	samples,
		QRhiTexture::Flags	= `{}`
	)

Create a render target from a texture format and size.

This function will also create a texture.

◆ makeCompute()

QShader score::gfx::makeCompute	(	const RenderState &	v,
		QString	compt
	)

Compile a compute shader.

Note: this function will throw if the shader is invalid.

◆ makeShaders()

std::pair< QShader, QShader > score::gfx::makeShaders	(	const RenderState &	v,
		QString	vert,
		QString	frag
	)

Get a pair of compiled vertex / fragment shaders from GLSL 4.5 sources.

Note: this function will throw if a shader is invalid.

Detailed Description

Classes

Typedefs

Enumerations

Functions

Variables

Function Documentation

◆ createRenderTarget()

◆ makeCompute()

◆ makeShaders()