[Flutter] Is there an efficient way of writting to a Flutter canvas/surface or Dart buffer...

An older, but developed and maintained C/C++ application draws its own graphical interface (with everything involved). An option to make it cross-platform is using Flutter, with a first focus on Android.

My problem is basically how to access and directly write to a Flutter canvas/surface or at least Dart buffer from native C/C++ code?

I have searched in a lot of directions, like (only listing the most relevant to me):

• Flutter GPU is too low-level. It and the derived Flutter Scene package are still in early preview (requiring some experimental features) and evidently target the GPU, which is not a fit for my need.
• The Video Player and Camera Flutter packages require a movie and respectively camera source.
• Creating an Android plugin for Flutter, like using the Java Surface Producer, isn't portable nor maintainable in my case.
• Mingling inside the Flutter rendering process, as some Alibaba Medium articles mention, is also out-of-scope for me.

So, the most straight-forward and simple solution seemed to be drawing on a Flutter Canvas, specifically drawing an image on it. I imaged that I could somehow get the buffer of an image, pass a pointer thereof to the native side and directly write into that, but I failed in doing this trying all possibilities that I could think or look for -- maybe with Native Assets I've been the closest to success, but my test app crashes at a point and it's cumbersome to debug on Windows for Android.

I did manage to succeed by creating a C buffer on the Dart side, passing its pointer to the native C/C++ side and writing into it on the native side, as this issue explains. But then I need to copy from the C buffer into a Dart array, and then again I need to create an image from that Dart array, which is lastly drawn on the canvas. So, instead of 1 write I need 3 writes (copying 3 times) in my test app, which you can image has a serious performance hit.

My test application basically loads 60 images (containing a visual counter) and just cycles through them, to roughly benchmark the raw rendering performance of Flutter on Android. Below are the most important code parts of my FFI Plugin project (let's call it flutter_render).

main.dart:

import 'dart:async';
import 'dart:ffi';
import 'dart:ui' as ui;

import 'package:ffi/ffi.dart';
import 'package:flutter/material.dart';
import 'package:flutter/services.dart';
import 'package:flutter_render/flutter_render.dart' as flutter_render;

import 'my_painter.dart';

void main() {
runApp(const MyApp());
}

class MyApp extends StatefulWidget {
const MyApp({super.key});

@override
State<MyApp> createState() => _MyAppState();
}

class _MyAppState extends State<MyApp> {
static const _kImageRelPath = "images/No_";
static const _kImageExt = ".png";
static const _kImageWidth = // ...
static const _kImageHeight = // ...
static const _kPixelFormat = ui.PixelFormat.rgba8888;
static const _kPixelSize = 4;
static const _kImageSize = _kImageWidth * _kImageHeight * _kPixelSize;

var imageIndex = 0;
ui.Image? currentImage;
//var images = <ui.Image>[];
late ByteData imageData;
late final Uint8List imageBytes;
late final Pointer<Uint8> imageCBuffer;
late Timer timer;

@override
void initState() {
super.initState();

initStateAsync();
}

@override
void dispose() {
super.dispose();

flutter_render.deleteImages();
malloc.free(imageCBuffer);
}

@override
Widget build(BuildContext context) {
return MaterialApp(
home: Container(
color: Colors.red,
child: CustomPaint(
painter: MyPainter(currentImage),
size: const ui.Size(_kImageWidth + .0, _kImageHeight + .0),
),
)
);
}

void initStateAsync() async {
imageCBuffer = malloc<Uint8>(_kImageSize);
flutter_render.setDartBuffer(imageCBuffer, _kImageSize);
// create imageBytes

// code to load the images from disc, if loaded from Dart, or else
flutter_render.loadImages();

timer = Timer.periodic(const Duration(milliseconds: 50/*20*/), handleTimeout);
}

void handleTimeout(Timer timer) {
//setState(() => _moveToNextImage());
setState(() => _getNextImage());
}

// void _moveToNextImage() {
// ++imageIndex;
// final len = images.length;
// if (len > 0) {
// imageIndex %= len;
// currentImage = images[imageIndex];
// }
// }

void _getNextImage() {
flutter_render.fillDartBuffer();
_copyCBufferToList(imageCBuffer, imageBytes);
ui.decodeImageFromPixels(imageCBuffer.asTypedList(_kImageSize),
_kImageWidth, _kImageHeight, _kPixelFormat,
(img) => currentImage = img);
}

void _copyCBufferToList(Pointer<Uint8> cBuffer, Uint8List list) {
var cb8 = cBuffer.asTypedList(list.length);
for (int i = 0; i < _kImageSize /*list.length*/; i++) {
list = cb8;
}
}
}


my_painter.dart:

import 'dart:ui' as ui;

import 'package:flutter/cupertino.dart';

class MyPainter extends CustomPainter {
final paintObj = Paint();
ui.Image? image;

MyPainter(this.image);

@override
void paint(Canvas canvas, Size size) {
if (image != null) {
canvas.drawImage(image!, const Offset(0, 0), paintObj);
}
}

@override
bool shouldRepaint(covariant CustomPainter oldDelegate) {
return true;
}
}


flutter_render.c:

#include "flutter_render.h"

#include <string.h>

#define MAX_PICTURES 60

static int imageCount = 0;
static int imageIndex = 0;
static uint8_t *images[MAX_PICTURES] = {
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
};
static uint8_t *dartBuffer = 0;
static int dartBufferSize = 0;

FFI_PLUGIN_EXPORT void loadImages() {
//code to load the images from disk
}

FFI_PLUGIN_EXPORT void deleteImages() {
for (int i = 0; i < imageCount; ++i) {
if (images != 0) free(images);
images = 0;
}
}

FFI_PLUGIN_EXPORT void setDartBuffer(uint8_t *buffer, int bufferSize) {
dartBuffer = buffer;
dartBufferSize = bufferSize;
}

FFI_PLUGIN_EXPORT void fillDartBuffer() {
if (dartBuffer == 0 || imageCount != MAX_PICTURES) return;

imageIndex %= MAX_PICTURES;
memcpy(dartBuffer, images[imageIndex], dartBufferSize);
++imageIndex;
}


On a low end Android device of interest I measured about 100 average fps from managed to managed (loading the images on the Dart side, see the commented Dart code, e.g. _moveToNextImage()), and respectively ~41 fps going from native to managed with the Skia engine. With the Impeller engine the fps was ~82 in the latter case, but there were frequent frame drops.

I'm open to solutions that take a different path. Thank you!

Continue reading...