[Flutter] Use flutter shader to create pixel painting software to improve...

I am writing drawing software with flutter, and in order to improve the performance of the software and add features such as brushes, I use shaders to draw. That is, I set the image to the shader before drawing, I have observed that it performs very well after this, and after drawing it uses toImageSync() to convert it into a ui.Image object for rendering to the screen

My current approach:

I currently have a Layer class which contains properties such as blending mode, hiding layers, and most importantly a Uint8List variable which stores the information of layers. Assuming my canvas size is 16 x 16, the information stored in a Uint8List variable is 16 x 16 x 4. There is also a variable of type ui.Image, which is used when blending multiple layers. ui.Image will only be updated if the Unint8List variable is modified. This is my current approach.

//isInvalidated The function of the variable is to mark whether the data of the current layer has been modified. If it has been modified, it is generated by executing ui.decodeImageFromPixels().

//image Image for layer blending

Future<ui.Image?> render(ui.Size canvasSize) async {

if (!isInvalidated) {
return image;
}
isInvalidated = false;

if (!isCanRender) {
return null;
}

var complater = Completer<ui.Image>();

ui.decodeImageFromPixels(pixelData, canvasSize.width.toInt(), canvasSize.height.toInt(), ui.PixelFormat.rgba8888, complater.complete);

image = await complater.future;

return image;
}




//This is the function used for layer blending
Future<ui.Picture> blendLayers(BuildContext context, Size canvasSize) async {
ui.PictureRecorder recorder = ui.PictureRecorder();
ui.Canvas canvas = ui.Canvas(recorder);

var frameModel = context.read<FrameModel>();

for (int i = 0; i < frameModel.frameList[frameModel.currentEditFrame].layers.length; i++) {
if (!frameModel.frameList[frameModel.currentEditFrame].layers.visible) {
continue;
}

final image = await frameModel.frameList[frameModel.currentEditFrame].layers.render(context.read<ToolParameter>().canvasSize);
var layerBlendMode = frameModel.frameList[frameModel.currentEditFrame].layers.blendMode;

if (i == 0) {
layerBlendMode = LayerBlendMode.normal;
}

if (image != null) {
canvas.drawImage(
image,
Offset.zero,
Paint()
..color = Color.fromRGBO(255, 255, 255, frameModel.frameList[frameModel.currentEditFrame].layers.transparence / 100.0)
..blendMode = getBlendMode(layerBlendMode));
}
}

return recorder.endRecording();
}


This method has achieved good performance, but I feel that it is not enough, and I have been trying new things so that I can provide better software to my users, which is also my dream.

I have tried:

I didn't have a lot of experience with shaders before, but to improve performance, I gave them a try, and I have to say that they did have good performance, but after a while there would be memory issues until they crashed.

#version 460 core

#include <flutter/runtime_effect.glsl>

precision mediump float;

//Output color
out vec4 fragColor;

//Center position and radius
uniform vec2 u_center;
uniform float u_radius;

//brush color
uniform vec4 u_color;

//shape type
uniform float u_shape_type;

//The texture is valid if the selected type is Custom
//uniform sampler2D uTexture;

//x-symmetry
uniform float x_axis_symmetry;

//x-symmetry position
//uniform float x_axis_symmetry_position;

//y-symmetry
//uniform float y_axis_symmetry;

//y-symmetry position
//uniform float y_axis_symmetry_position;



void main()
{
if(u_shape_type == 0){
//draw rect
if(FlutterFragCoord().xy.x > u_center[0] - u_radius &&
FlutterFragCoord().xy.x < u_center[0] + u_radius &&
FlutterFragCoord().xy.y > u_center[1] - u_radius &&
FlutterFragCoord().xy.y < u_center[1] + u_radius)
{
fragColor = u_color;
}else{
fragColor = vec4(0.0);
}
}else if(u_shape_type == 1){
//Calculate the distance from the current coordinate to the center of the circle
float dis = distance(FlutterFragCoord().xy, u_center);
//If the distance is less than the radius, color is drawn, otherwise transparent
if (dis < u_radius)
{
fragColor = u_color;
}
else
{
fragColor = vec4(0.0);
}
}else if(u_shape_type == 2){
//chartlet
//fragColor = texture(u_texture, FlutterFragCoord().xy / u_resolution);
}
}


This is my shader code

ui.PictureRecorder recorder = ui.PictureRecorder();
ui.Canvas canvas = ui.Canvas(recorder);

//Create a shader and set shader parameters
//code ...

//canvas.draw()...
//code ...

ui.Image image = recorder.endRecording().toImageSync(canvas_width, canvas_height);


Then use image to blend the layers

I expect results :

In my observations, performance did improve, but software memory was not smooth.

What I expect is that the software memory can be smooth, while the performance can be improved

Continue reading...