React Native MasteryThe Ultimate React Native and Expo Course 🚀

React Native Skia Video and 3d API with WebGPU

Anisur Rahman profile picture
Anisur RahmanJun 8, 2024

The React Native Skia team released V1.0 in March this year to complete the drawing features by providing the Paragraph API & Atlas API. After that release, the team felt that React Native Skia has the best drawing capabilities, but it should also be able to be used with any data source (e.g., Image, Video, 3D projections, etc).

React Native Skia Video

React Native Skia already provides a feature where images can be rendered using a shader. A shader is a small program that runs on the GPU (Graphics Processing Unit) and instructs the GPU on how to draw each pixel on the screen. Shaders in Skia are written using Skia’s shading language, similar to GLSL (OpenGL Shading Language). Below is an example of image shading using the React Native Skia Image Shaders API.

react-native-skia-image.gif

Did you notice that the image pixels were re-rendered when we slid the image from right to left? Yes, that’s the effect of the Image Shaders API.

However, this shader was not provided with the video. Therefore, William announced that from now on, React Native Skia can be applied to videos too. Thus, the static image shader examples above can now be applied to videos, and shaders from React Native Skia can be used on videos as shown below.

embed

It’s pretty cool, right? From now on, it’s also possible to apply filters and effects to videos using React Native Skia, just like the ones shown below.

react-native-skia-video-filters.gif

Now that we can apply shaders to videos as well, it opens the door for Ambient Mode (soft glow of colors from the video onto the surroundings of the video) while playing a Video.

react-native-skia-ambient.gif

The Native Buffer API

The above React Native Skia Video was developed using a new API named Native Buffers, created through the collaboration of William & Margelo team, led by Marc. It uses platform-specific GPU APIs (Metal for iOS & OpenGL for Android) which enable direct GPU access and data processing for efficient graphics renderingusing the following memory management classes from different Native SDKs.

  • Android uses the "HardwareBuffer" class
  • iOS uses the "PixelBuffer" class
  • Web uses the "CanvasImageSource" class

native-buffers.png

NOTE: React Native Vision Camera is now using React Native Skia; however, its use is optional. This means you can use the React Native Skia GPU feature only when necessary.

WebGPU: 3D API with React Native Skia

Previously, a significant portion of 3D animation tasks depended heavily on the CPU. However, these have now been transitioned to the GPU, with examples including 3D transformations (Ex: Rotating). Below is the current architecture of React Native Skia.

react-native-skia-arch.png

Here, you can see that Skia runs on top of the OpenGL GPU API for Android and the Metal GPU API for iOS. However, have you noticed that this approach wastes a lot of time because the OpenGL GPU code will not work for Metal GPU, and vice versa? Consequently, the React Native Skia team has planned to modernize this architecture.

The plan is to implement a unified 3D API (‘WebGPU’) across both iOS and Android. Skia will operate on top of this unified API ‘WebGPU’. This will enable the use of libraries such as Three.js (used to create and display animated 3D computer graphics) on top of Skia.

Three reasons why ‘WebGPU’ was chosen as the backend for Skia.

  1. ⁠It modernized the React Native Skia backend API
  2. ⁠⁠It creates a unified 3D API across IOS and Android
  3. It has the general computing capabilities to run the operations in GPU instead of CPU. Ex: To run an AI model needs high GPU capabilities.

Below is the new architecture of React Native Skia.

react-native-skia-new-arch.png

Let’s understand each point of this new architecture of React Native Skia 🚀

  • Vulkan: It is the new & modern GPU API for Android. Previously the team was using OpenGL but it is not supported in the newest Skia backend.
  • Metal: Metal is already a modern GPU API for iOS & the team is using it from old architecture.
  • Dawn WebGPU: It is the C++ implementation of Google ‘WebGPU’. It reduced 1000 lines of code (by Vulkan or Metal) to 20 lines for a triangle to draw.
  • Skia Graphite: It is the new modern backend for modern GPU APIs (Ex: Metal, Vulkan, etc). It is a high-level graphics library that focuses on rendering with support for GPU acceleration (offloading compute-intensive tasks from the CPU to the GPU) through Vulkan and other backends.

That’s it 🙌

React Native Skia is pushing the boundaries not only for 2d Drawings, but now also for 3d. This is going to open the doors to a lot of new possibilities in React Native.

Did you learn something new today?

If you found this post valuable, share it with one friend or coworker that can benefit from it as well. That would be much appreciated 🙏

The post was written by Anis and edited by Vadim Savin.


Anisur Rahman profile picture

Anisur Rahman

Sr. React Native Engineer at BasementSports.

Author of "RN Advanced Book" (1.3K+)


Read next