Hey notJust Developers,

A few months ago, Software Mansion organized App JS Conf 2024. The exciting news from that conference is that the Infinite Red team has integrated on-device machine learning into React Native using Google ML Kit. Today, we will explore Gant Laborde’s presentation at the conference on the newly created React Native MLKit package.

NOTE: If you want to learn more about all the exciting talks at App JS Conf 2024, please take a look at notJust.dev/blog.

This issue is sponsored by RevenueCat​

In-app subscriptions are a pain. The code can be hard to write, time-consuming to maintain, and full of edge cases. RevenueCat makes it simple, plus provides analytics, integrations, and tools to grow so you can get back to building your app.

​Get started with RevenueCat for free, and save weeks of development time integrating In-app subscriptions.

Before the Google ML Kit

To understand the importance of Google ML Kit, let’s explore a common scenario from our daily lives: document scanning. Imagine an insurance company app where users can upload images of documents (such as bank checks or cards) for the operational team to extract text and save it for further review.

Now, consider a situation where a user uploaded the image above (the card), and the operational team used online OCR (Optical Character Recognition) tools like below to parse the text from the card.

Oops! OCR shows us “No recognized text”! But why? Did you notice that the image was taken at a right angle? Yeah, that's the reason why the online OCR tool can’t detect the image. So how fast can the operational team fix the angle issue of that image? One possible straightforward solution could be Photoshop. Let’s try 🤷‍♂️.

After editing in Photoshop, the image of the card now looks like below.

Now, let’s upload this new image of the card into that online OCR tool again.

Wow, now it has successfully parsed all the text from that image of the card! Isn’t it cool? But one point is ongoing in our minds, right? Yeah, that is “How can we automate that Photoshop process mentioned above?”. Specifically, how can we ensure that images are captured (or can be edited with one click) at the correct angle, no matter how they’re taken? (You can think for a moment 🤔).

The solution instead of Photoshop

The solution is in mathematics! By using complex mathematical concepts such as linear algebra, linear transformations, and 3D/4D matrices, you can adjust photo angles directly on your phone with just a single click. No more need for Photoshop! Ahh, but developers don’t have enough time to do such rocket science with a deadline! So, what’s the alternative 🤔?

Google ML Kit in action 💯

This is where the Google ML Kit package comes into play. It removed the workload for developers dealing with deep linear algebra, linear transformations, and more.

What is Google ML Kit?

​Google ML Kit is a mobile SDK (Software Development Kit) that brings Google’s machine-learning expertise to Android and iOS apps. It runs directly on users’ devices. This means that it moves the machine learning model into your phone, so you don’t have to worry about the internet anymore. You also don’t need to wait for any remote server to stream down the result to your phone. It only consumes your battery, without any server costs. Additionally, it protects your privacy as it works offline.

Short brief on ML & AI

Hey, some of you may be worried about ML (Machine Learning) and AI (Artificial Intelligence) here. Let’s take a short intro on these two. Machine learning is a field (a part) of artificial intelligence (AI) that focuses on creating algorithms and models that are capable of learning from other data and making predictions based on examples. Some other parts of AI include:

• Deep Learning: A specialized form of ML that can solve complex tasks.
• Robotics: It involves creating intelligent machines capable of interacting with their environment.
• Neural Networks: It is the backbone of deep learning algorithms.

Features of Google ML Kit

Now, let’s take a look at all the features that are available in the Google ML Kit SDK.

• Face Detection: Identifies and locates faces in images.
• Barcode Scanning: Reads and processes barcodes (QR code, Linear format, etc).
• Face Mesh: Detects detailed facial features (Ex: Location & shape of Eyes, Nose, etc).
• Text Recognition: Extracts text from images.
• Image Labeling: Identifies objects, locations, and activities from images.
• Object Tracking: Locates and tracks objects in real time directly from the camera.
• Digital Ink: Identifies handwritten texts.
• Pose Detection: Determines the position of the human body parts.
• Selfie Segmentation: Separates users from the background in scenes.
• Subject Segmentation: Isolates subjects (people, pets, or objects) from picture backgrounds.
• Document Scanning: Detects document edges from the image & enhances quality. This feature can solve our above issue of Photoshop 💯.

Google ML Kit in React Native 🔥

So, Google ML Kit has lots of amazing features! But when will Google ML Kit be available in React Native? The good news is that the @InfiniteRed team has already done it and published a library called React Native MLKit, which can even be used in Expo apps. Currently, it supports all the following features.

• Face Detection
• Object Detection
• Image Labeling
• Document Scanning

NOTE: The team is working hard to onboard all the other features in React Native that are available in the original Google ML Kit SDK. Additionally, the team has made the repository open-source. You can contribute even by giving the repo a STAR ⭐️. Kudos to all members of the team, especially Jamon, Gant Laborde, Frank Calise & Mazen Chami.

Code with React Native MLKit

Now it is time to do some code. The team has published separate NPM packages for each of the modules mentioned above because they wanted to give developers the option to include only the modules they need in their app. All the common classes and logic are inside the “@infinitered/react-native-mlkit-core” package. Below, you’ll find the installation commands for both Yarn and npm.

# Face Detection #
# yarn
yarn add @infinitered/react-native-mlkit-face-detection
# npm
npm install @infinitered/react-native-mlkit-face-detection

# Image Labeling #
# yarn
yarn add @infinitered/react-native-mlkit-image-labeling
# npm
npm install @infinitered/react-native-mlkit-image-labeling

# Object Detection #
# yarn
yarn add @infinitered/react-native-mlkit-object-detection
# npm
npm install @infinitered/react-native-mlkit-object-detection

# Document Scanner #
# yarn
yarn add @infinitered/react-native-mlkit-document-scanner
# npm
npm install @infinitered/react-native-mlkit-document-scanner

Code for Face Detection feature

Let’s take a look at a code example for the Face Detection feature. First, install the “Face Detection” package using the command above, and then wrap the entire app with the “Face Detection” context provider as shown below.

/// App.tsx

import { RNMLKitFaceDetectionContextProvider } from "@infinitered/react-native-mlkit-face-detection";

function App() {
  return (
    <RNMLKitFaceDetectionContextProvider>
      {/* rest of your app goes here */}
    </RNMLKitFaceDetectionContextProvider>
  );
}

Now, you can detect faces in any photo by using the useFacesInPhoto hook, which only requires the image’s URI, as shown in the code snippet below. The hook will return three values.

1. faces: It is an array of objects where each object contains details of each face on the image (e.g., face X-axis angle, Y-axis angle, etc.).
2. error: It represents error details.
3. status: It shows the status while detecting faces from an image (e.g., ready, detecting, done, etc.).

/// FaceDetectionScreen.tsx

import { useFacesInPhoto } from "@infinitered/react-native-mlkit-face-detection";

function FaceDetectionComponent() {
  const { faces, error, status } = useFacesInPhoto(
    "local_uri_of_your_image_uri",
  );

  if (error) {
    return <Text>Error: {error}</Text>;
  }

  return (
    <View>
      {faces.map((face) => (
        <View key={face.trackingId}>
          <Text>{JSON.stringify(face)}</Text>
        </View>
      ))}
    </View>
  );
}

Now, if you upload an image from your gallery (or take one instantly) with two faces (human face) and log the value of the faces array and status in the above code, you will see an output like below.

You’re almost there. Now that you have the face details, you just need to display the image on the screen with detected faces highlighted by green square boxes like below. To do that, pass the faces array as props to the <ImageWithBoundingBoxes/> component, which is imported from the “@infinitered/react-native-mlkit-core” package. You can find the full code example here.

Demo Project is there 💯

The team has also created a React Native app using Expo and published it on GitHub for developers to explore the code implementation of all the available features of React Native MLKit. Follow the steps below to run it on your emulator or Android devices. We’ve shown here the setup for Android only, but you can also run the project on an iOS device by following this link.

First, clone the project using the command below in your terminal.

git clone git@github.com:infinitered/react-native-mlkit.git

Now install all dependencies of the project.

cd react-native-mlkit
yarn install

Then you need to build the native modules by this below command. It may take some time.

yarn build

After the build, go to the ~apps/ExampleApp directory & run the below command.

yarn android

After successfully running the app, you will see screens like the ones below, where you can explore & update the code of all available features provided by React Native MLKit SDK.

Live App on Stores 🚀

The team has surprisingly published the demo app on both the Google Play Store and App Store. You can download the app instantly and explore all the available features of React Native MLKit.

That’s it 🙌

React Native MLKit is a native module for Expo and React Native that allows you to use the Google MLKit library in your app, with available modules for Face Detection, Object Detection, Image Labeling, and Document Scanner. 🚀

🔁 In case you missed it

Building the iOS 18 Photos App with React Native and Reanimated In this video, we'll be building the iOS 18 Photos app clone using React Native, Expo, and Reanimated. We'll explore how to use these powerful tools to create stunning animations and user interactions. Watch now

Build an AI-Powered Google Translate Clone with React Native and Expo In this tutorial, we will be building an AI-powered Google Translate clone using React Native and Expo. You'll learn to integrate OpenAI for text translations, speech-to-text, and text-to-speech functionalities. Watch now

Did you learn something new today?

If you found this email valuable, forward it to one friend or coworker who can benefit from it as well. That would be much appreciated 🙏

The newsletter was written by Anis and edited by Vadim Savin.

Vadim Savin Helping you become a better developer together with the notJust.dev team