score::gfx Namespace Reference

Graphics rendering pipeline for ossia score. More...

Detailed Description

Graphics rendering pipeline for ossia score.

This namespace provides a rendering pipeline for GPU visuals, by the way of a rendering graph. The implementation uses the Qt RHI:

https://www.qt.io/blog/graphics-in-qt-6.0-qrhi-qt-quick-qt-quick-3d

It is mainly designed for rendering a succession of visual effects, not rendering arbitrary 3D. Shaders are written in the usual glslang -> spirv-cross fashion, with GLSL #version 450. They are then translated automatically to SPIRV, Metal, GLSL or D3D11 shaders.

The overall design is as follows :

• Elements to render are represented as nodes to a graph: score::gfx::Node
• Input ports are the uniforms and input textures & audio data.
• Output ports generally are just the texture we are rendering to.
• score::gfx::Graph contains all the nodes, and relationships between these nodes.
• For each output sink (a window surface, an NDI or Spout output...), a score::gfx::RenderList is created. It contains all the nodes to render for a given output.
• For each score::gfx::Node, the score::gfx::RenderList creates a score::gfx::NodeRenderer which contains the actual GPU resource handles for the sink.

Given the following graph :

[ video ] -> [ shader 1 ] -> [ screen 1 ]
                          \
                           -> [ shader 2] -> [ Spout output ]

Two render lists are created, the first being

[ video ] -> [ shader 1 ] -> [ screen 1 ]

and the second being

[ video ] -> [ shader 1 ] -> [ shader 2] -> [ Spout output ]

Rendering for these two RenderList is entirely independent - they can render at different sizes and potentially use different graphic APIs, e.g. one OpenGL and the other Vulkan. This is also done to enable threaded rendering for each of these.

We provide the most essential nodes for building interesting visual effects:

• A window output node (score::gfx::ScreenNode).
• Shader filters using Interactive Shader Format (https://isf.video).
• Video (score::gfx::VideoNode) and image (score::gfx::ImagesNode) renderers.
• The video renderer tries to do as much of the decoding as possible on the GPU: for instance YUV420, HAP, ... are decoded on the GPU. Otherwises it falls back on libavcodec to guarantee compatibility with a wide range of video formats.

Other plug-ins may provide other useful things, such as NDI I/O or particle effects.

See GfxPlugins for information on how to create your custom nodes, visual effects and video decoders.

Classes
struct	AudioTexture
	Data model for audio data being sent to the GPU. More...
struct	AudioTextureUpload
struct	BasicMesh
class	CustomMesh
struct	DefaultShaderMaterial
	Utility to represent a shader material following score conventions. More...
struct	DepthNode
struct	Edge
	Connection between two score::gfx::Port. More...
struct	EmptyDecoder
	Default decoder when we do not know what to render. More...
struct	FullScreenImageNode
class	GenericNodeRenderer
	Generic renderer. More...
struct	geometry
struct	geometry_input_port_vis
class	GeometryFilterNode
	Data model for geometry filters. More...
struct	GeometryFilterNodeRenderer
class	GPUVideoDecoder
	Processes and renders a video frame on the GPU. More...
struct	Graph
	Represents a graph of renderers. More...
struct	HAPDecoder
	Base class for HAP ((c) Vidvox) decoding. More...
struct	HAPDefaultDecoder
	Decodes HAP basic format. More...
struct	HAPMDecoder
	Decodes HAP-M (HAP + alpha channel) More...
struct	image
struct	Image
	Image data and metadata. More...
struct	ImagesNode
	A node that renders an image to screen. More...
struct	isf_input_port_vis
struct	isf_input_size_vis
class	ISFNode
	Data model for Interactive Shader Format filters. More...
struct	Mesh
	Data model for meshes. More...
struct	MeshBuffers
struct	Message
struct	ModelCameraUBO
	UBO shared across all entities shown with the same camera. More...
struct	ModelDisplayNode
	A node that renders a model to screen. More...
class	Node
	Root data model for visual nodes. More...
class	NodeModel
	Common base class for most single-pass, simple nodes. More...
class	NodeRenderer
	Renderer for a given node. More...
struct	NV12Decoder
	Decodes NV12 videos. More...
class	OutputNode
	Base class for sink nodes (QWindow, spout, syphon, NDI output, ...) More...
class	OutputNodeRenderer
struct	OutputUBO
	UBO shared across all entities shown on the same output. More...
struct	PackedDecoder
struct	PackedRectDecoder
struct	Pass
struct	PersistSampler
struct	PhongNode
struct	Pipeline
	Useful abstraction for storing a graphics pipeline and associated resource bindings. More...
struct	PlainMesh
	A mesh with only position attributes. More...
struct	PlainTriangle
	A triangle mesh with only positions. More...
struct	PlanarDecoder
struct	Port
	Port of a score::gfx::Node. More...
class	ProcessNode
	Common base class for nodes that map to score processes. More...
struct	ProcessUBO
	UBO specific to individual processes / nodes. More...
struct	RefcountedFrame
struct	RenderedISFNode
struct	RenderedPhongNode
class	RenderList
	List of nodes to be rendered to an output. More...
struct	RenderState
	Global state associated to a rendering context. More...
struct	RGB24Decoder
struct	Sampler
	Stores a sampler and the texture currently associated with it. More...
struct	ScreenNode
	This node is used for rendering to a score::gfx::Window. More...
struct	ShaderCache
	Cache of baked QShader instances. More...
struct	SimpleRenderedISFNode
struct	TexgenNode
struct	TextNode
	A node that renders text to screen. More...
struct	TexturedMesh
	A mesh with positions and texture coordinates. More...
struct	TexturedQuad
	A quad mesh with positions and texture coordinates. More...
struct	TexturedTriangle
	A triangle mesh with positions and texture coordinates. More...
struct	TextureRenderTarget
	Useful abstraction for storing all the data related to a render target. More...
struct	Timings
	Messages sent from the execution thread to the rendering thread. More...
struct	UYVY422Decoder
	Decodes UYVY422 video, mostly used for NDI. More...
struct	VideoFrameReader
struct	VideoFrameShare
struct	VideoMaterialUBO
	UBO shared across all video objects. More...
class	VideoNode
	Model for rendering a video. More...
class	VideoNodeBase
class	VideoNodeRenderer
class	Window
	A platform window in which the content is going to be rendered. More...
struct	YUV420Decoder
	Decodes YUV420 videos. More...
struct	YUV420P10Decoder
	Decodes YUV420 videos. More...
struct	YUV420P12Decoder
	Decodes YUV420 videos. More...
struct	YUV422Decoder
	Decodes YUV422 videos. More...
struct	YUV422P10Decoder
	Decodes YUV422 videos. More...
struct	YUV422P12Decoder
	Decodes YUV422 videos. More...
struct	YUYV422Decoder
	Decodes YUYV422 video. More...

Typedefs
using	Vertex = score::gfx::Node *
using	GraphImpl = boost::adjacency_list< boost::vecS, boost::vecS, boost::directedS, Vertex >
using	VertexMap = ossia::hash_map< score::gfx::Node *, GraphImpl::vertex_descriptor >
using	FunctionMessage = std::function< void(score::gfx::Node &)>
using	gfx_input = ossia::slow_variant< ossia::monostate, ossia::value, ossia::audio_vector, ossia::geometry_spec, ossia::transform3d, FunctionMessage >
using	PassMap = ossia::small_vector< std::pair< Edge *, Pipeline >, 2 >
using	PassOutput = ossia::variant< PersistSampler, TextureRenderTarget >

Enumerations
enum	ImageMode { Single , Clamped , Tiled , Mirrored }
enum	GraphicsApi {   Null , OpenGL , Vulkan , D3D11 ,   Metal }
	Available graphics APIs to use.
enum	ScaleMode { Original , BlackBars , Fill , Stretch }
	How to resize a texture to adapt it to a viewport.
enum class	Types {   Empty , Int , Float , Vec2 ,   Vec3 , Vec4 , Image , Audio ,   Camera , Geometry , Buffer }

Functions
template<typename Graph_T , typename IO >
void	print_graph (Graph_T &g, IO &stream)
	Q_DECLARE_OPERATORS_FOR_FLAGS (Mesh::Flags)
void	defaultPassesInit (PassMap &passes, const std::vector< Edge * > &edges, RenderList &renderer, const Mesh &mesh, const QShader &v, const QShader &f, QRhiBuffer *processUBO, QRhiBuffer *matUBO, const std::vector< Sampler > &samplers, std::span< QRhiShaderResourceBinding > additionalBindings)
void	defaultRenderPass (RenderList &renderer, const Mesh &mesh, const MeshBuffers &bufs, QRhiCommandBuffer &cb, Edge &edge, PassMap &passes)
void	quadRenderPass (RenderList &renderer, const MeshBuffers &bufs, QRhiCommandBuffer &cb, Edge &edge, PassMap &passes)
TextureRenderTarget	createRenderTarget (const RenderState &state, QRhiTexture *tex, int samples)
	Create a render target from a texture.
TextureRenderTarget	createRenderTarget (const RenderState &state, QRhiTexture::Format fmt, QSize sz, int samples, QRhiTexture::Flags={})
	Create a render target from a texture format and size.
void	replaceBuffer (std::vector< QRhiShaderResourceBinding > &tmp, int binding, QRhiBuffer *newBuffer)
void	replaceSampler (std::vector< QRhiShaderResourceBinding > &tmp, int binding, QRhiSampler *newSampler)
void	replaceTexture (std::vector< QRhiShaderResourceBinding > &tmp, int binding, QRhiTexture *newTexture)
void	replaceBuffer (QRhiShaderResourceBindings &srb, int binding, QRhiBuffer *newBuffer)
void	replaceSampler (QRhiShaderResourceBindings &srb, int binding, QRhiSampler *newSampler)
void	replaceTexture (QRhiShaderResourceBindings &srb, int binding, QRhiTexture *newTexture)
void	replaceSampler (QRhiShaderResourceBindings &, QRhiSampler *oldSampler, QRhiSampler *newSampler)
	Replace a sampler.
void	replaceTexture (QRhiShaderResourceBindings &, QRhiSampler *sampler, QRhiTexture *newTexture)
	Replace the texture currently bound to a sampler.
void	replaceTexture (QRhiShaderResourceBindings &srb, QRhiTexture *old_tex, QRhiTexture *new_tex)
	Replace a texture by another in a set of bindings.
Pipeline	buildPipeline (const RenderList &renderer, const Mesh &mesh, const QShader &vertexS, const QShader &fragmentS, const TextureRenderTarget &rt, QRhiShaderResourceBindings *srb)
QRhiShaderResourceBindings *	createDefaultBindings (const RenderList &renderer, const TextureRenderTarget &rt, QRhiBuffer *processUBO, QRhiBuffer *materialUBO, const std::vector< Sampler > &samplers, std::span< QRhiShaderResourceBinding > additionalBindings={})
	Create bindings following the score conventions for shaders and materials.
Pipeline	buildPipeline (const RenderList &renderer, const Mesh &mesh, const QShader &vertexS, const QShader &fragmentS, const TextureRenderTarget &rt, QRhiBuffer *processUBO, QRhiBuffer *materialUBO, const std::vector< Sampler > &samplers, std::span< QRhiShaderResourceBinding > additionalBindings={})
	Create a render pipeline following the score conventions for shaders and materials.
std::pair< QShader, QShader >	makeShaders (const RenderState &v, QString vert, QString frag)
	Get a pair of compiled vertex / fragment shaders from GLSL 4.5 sources.
QShader	makeCompute (const RenderState &v, QString compt)
	Compile a compute shader.
QSize	resizeTextureSize (QSize img, int min, int max) noexcept
	Resize the size of a texture to fit within GPU limits.
QImage	resizeTexture (const QImage &img, int min, int max) noexcept
	Resize a texture to fit within GPU limits.
QSizeF	computeScale (score::gfx::ScaleMode mode, QSizeF viewport, QSizeF texture)
	Comput the scale to apply to a texture so that it fits in a GL viewport.
void	copyMatrix (const QMatrix4x4 &mat, float *ptr) noexcept
void	copyMatrix (const QMatrix3x3 &mat, float *ptr) noexcept

Variables
const char *	frag

Function Documentation

createRenderTarget()

TextureRenderTarget score::gfx::createRenderTarget	(	const RenderState &	state,
		QRhiTexture::Format	fmt,
		QSize	sz,
		int	samples,
		QRhiTexture::Flags	= {}
	)

Create a render target from a texture format and size.

This function will also create a texture.

makeCompute()

QShader score::gfx::makeCompute	(	const RenderState &	v,
		QString	compt
	)

Compile a compute shader.

Note: this function will throw if the shader is invalid.

makeShaders()

std::pair< QShader, QShader > score::gfx::makeShaders	(	const RenderState &	v,
		QString	vert,
		QString	frag
	)

Get a pair of compiled vertex / fragment shaders from GLSL 4.5 sources.

Note: this function will throw if a shader is invalid.