Label images with a custom model on Android

Page Summary

outlined_flag

• ML Kit provides two ways to integrate custom image labeling models: bundled (in-app) and unbundled (downloaded).

• Custom models can be bundled within the app or hosted on Firebase Machine Learning for dynamic updates.

• The library offers various options for loading images, including from files, bitmaps, and camera streams.

• When using hosted models, ensure they are downloaded before processing images by using the isModelDownloaded() method.

• For optimal performance, consider input image format and rotation, and throttle calls to the image labeler in real-time applications.

There are two ways to integrate image labeling with custom models: by bundling the pipeline as part of your app, or by using an unbundled pipeline that depends on Google Play services. If you select the unbundled pipeline, your app will be smaller. See the following table for details.

There are two ways to integrate a custom model: bundle the model by putting it inside your app's asset folder, or dynamically download it from Firebase. The following table compares these two options.

Try it out

• See the vision quickstart app for an example usage of the bundled model and the automl quickstart app for an example usage of the hosted model.

Before you begin

1. In your project-level build.gradle.kts file, make sure to include Google's Maven repository in both your buildscript and allprojects sections.

2. Add the dependencies for the ML Kit Android libraries to your module's app-level gradle file, which is usually app/build.gradle.kts. Choose one of the following dependencies based on your needs:

   For bundling the pipeline with your app:

   dependencies {
     // ...
     // Use this dependency to bundle the pipeline with your app
     implementation("com.google.mlkit:image-labeling-custom:17.0.3")
   }
   

   For using the pipeline in Google Play services:

   dependencies {
     // ...
     // Use this dependency to use the dynamically downloaded pipeline in Google Play services
     implementation("com.google.android.gms:play-services-mlkit-image-labeling-custom:16.0.0-beta5")
   }
   

3. If you choose to use the pipeline in Google Play services, you can configure your app to automatically download the pipeline to the device after your app is installed from the Play Store. To do so, add the following declaration to your app's AndroidManifest.xml file:

   <application ...>
       ...
       <meta-data
           android:name="com.google.mlkit.vision.DEPENDENCIES"
           android:value="custom_ica" />
       <!-- To use multiple downloads: android:value="custom_ica,download2,download3" -->
   </application>
   

   You can also explicitly check the pipeline availability and request download through Google Play services ModuleInstallClient API.

   If you don't enable install-time pipeline downloads or request explicit download, the pipeline is downloaded the first time you run the labeler. Requests you make before the download has completed produce no results.

4. If you want to download a model using Cloud Storage for Firebase, make sure you add Firebase to your Android project, if you have not already done so. This is not required when you bundle the model.

1. Load the model

You can load the model from a locally-bundled source or a remotely-hosted source.

Configure a local model source

To bundle the model with your app:

1. Copy the model file (usually ending in .tflite or .lite) to your app's assets/ folder. (You might need to create the folder first by right-clicking the app/ folder, then clicking New > Folder > Assets Folder.)

2. Create LocalModel object, specifying the path to the model file:

   Kotlin

   val localModel = LocalModel.Builder()
           .setAssetFilePath("model.tflite")
           // or .setAbsoluteFilePath(absolute path to model file)
           // or .setUri(URI to model file)
           .build()

   Java

   LocalModel localModel =
       new LocalModel.Builder()
           .setAssetFilePath("model.tflite")
           // or .setAbsoluteFilePath(absolute path to model file)
           // or .setUri(URI to model file)
           .build();

Configure a remotely-hosted model source

To use the remotely-hosted model, you must download the model file to the device's local storage using your own app logic, and then load it as a local model. We recommend using Cloud Storage for Firebase to host a model. For implementation details, see the Firebase ML to Cloud Storage migration guide.

Configure the image labeler

After you configure your model sources, create an ImageLabeler object from one of them.

The following options are available:

If you only have a locally-bundled model, just create a labeler from your LocalModel object:

val customImageLabelerOptions = CustomImageLabelerOptions.Builder(localModel)
    .setConfidenceThreshold(0.5f)
    .setMaxResultCount(5)
    .build()
val labeler = ImageLabeling.getClient(customImageLabelerOptions)

CustomImageLabelerOptions customImageLabelerOptions =
        new CustomImageLabelerOptions.Builder(localModel)
            .setConfidenceThreshold(0.5f)
            .setMaxResultCount(5)
            .build();
ImageLabeler labeler = ImageLabeling.getClient(customImageLabelerOptions);