Getting on the same page with Paging 3

Posted by Florina Muntenescu, Android Developer Advocate

The Paging library enables you to load large sets of data gradually and gracefully, reducing network usage and system resources. You told us that the Paging 2.0 API was not enough - that you wanted easier error handling, more flexibility to implement list transformations like map or filter, and support for list separators, headers, and footers. So we launched Paging 3.0, now stable, a complete rewrite of the library using Kotlin coroutines (still supporting Java users) and offering the features you asked for.

Paging 3 highlights

The Paging 3 API provides support for common functionality that you would otherwise need to implement yourself when loading data in pages:

• Keeps track of the keys to be used for retrieving the next and previous page.
• Automatically requests the correct next page when the user scrolls to the end of the loaded data.
• Ensures that multiple requests aren’t triggered at the same time.
• Tracks loading state and allows you to display it in a RecyclerView list item or elsewhere in your UI, and provides easy retry functionality for failed loads.
• Enables common operations like map or filter on the list to be displayed, independently of whether you’re using Flow, LiveData, or RxJava Flowable or Observable.
• Provides an easy way of implementing list separators.
• Simplifies data caching, ensuring that you’re not executing data transformations at every configuration change.

We also made several Paging 3 components backwards compatible with Paging 2.0; so if you already use Paging in your app, you can migrate incrementally.

Adopting Paging 3 in your app

Let’s say that we’re implementing an app that displays all the good doggos. We get the doggos from a GoodDoggos API that supports index-based pagination. Let’s go over the Paging components we need to implement and how they fit into your app architecture. The following examples will be in Kotlin, using coroutines. For examples in the Java programming language using LiveData/RxJava, check out the documentation.

The Paging library integrates directly into the recommended Android app architecture in each layer of your app:

Paging components and their integration in the app architecture"

Defining the data source

Depending on where you’re loading data from, implement only a PagingSource or a PagingSource and a RemoteMediator:

• If you’re loading data from a single source, like network, local database, a file, etc, implement the PagingSource (if you’re using Room, it implements the PagingSource for you starting in Room 2.3.0-alpha).
• If you’re loading data from a layered source, like a network data source with a local database cache, implement the RemoteMediator to merge the two sources and a PagingSource for the local database cache.

PagingSource

A PagingSource defines the source of paging data and how to retrieve data from that single source. The PagingSource should be part of the repository layer. Implement load() to retrieve paged data from your data source and return the loaded data together with information about next and previous keys. This is a suspend function, so you can call other suspend functions here, such as the network call:

class DoggosRemotePagingSource(
    val backend: GoodDoggosService
) : PagingSource<Int, Dog>() {
  override suspend fun load(
    params: LoadParams<Int>
  ): LoadResult<Int, Dog> {
    try {
      // Load page 1 if undefined.
      val nextPageNumber = params.key ?: 1
      val response = backend.getDoggos(nextPageNumber)
      return LoadResult.Page(
        data = response.doggos,
        prevKey = null, // Only paging forward.
        nextKey = response.nextPageNumber + 1
      )
    } catch (e: Exception) {
        // Handle errors in this block
        return LoadResult.Error(exception)
    }
  }
}

PagingData and Pager

The container for paginated data is called PagingData. A new instance of PagingData is created every time your data is refreshed. To build a stream of PagingData create a Pager instance, using a PagingConfig configuration object and a function that tells the Pager how to get an instance of your PagingSource implementation.

In your ViewModel you construct the Pager object and expose a Flow<PagingData> to the UI. Flow<PagingData> has a handy cachedIn() method that makes the data stream shareable and allows you to cache the content of a Flow<PagingData> in a CoroutineScope. That way if you implement any transformations on the data stream, they will not be triggered again each time you collect the flow after Activity recreation. The caching should be done as close to the UI layer as possible, but not in the UI layer, as we want to make sure it persists beyond configuration change. The best place for this would be in a ViewModel, using the viewModelScope:

val doggosPagingFlow = Pager(PagingConfig(pageSize = 10)) {
  DogRemotePagingSource(goodDoggosService)
}.flow.cachedIn(viewModelScope)

PagingDataAdapter

To connect a RecyclerView to the PagingData, implement a PagingDataAdapter:

class DogAdapter(diffCallback: DiffUtil.ItemCallback<Dog>) :
  PagingDataAdapter<Dog, DogViewHolder>(diffCallback) {
  override fun onCreateViewHolder(
    parent: ViewGroup,
    viewType: Int
  ): DogViewHolder {
    return DogViewHolder(parent)
  }

  override fun onBindViewHolder(holder: DogViewHolder, position: Int) {
    val item = getItem(position)
    if(item == null) {
      holder.bindPlaceholder()
    } else {
      holder.bind(item)
    }
  }
}

Then, in your Activity/Fragment you’ll have to collect the Flow<PagingData> and submit it to the PagingDataAdapter. This is what the implementation would look like in an Activity onCreate():

val viewModel by viewModels<DoggosViewModel>()

val pagingAdapter = DogAdapter(DogComparator)
val recyclerView = findViewById<RecyclerView>(R.id.recycler_view)
recyclerView.adapter = pagingAdapter

lifecycleScope.launch {
  viewModel.doggosPagingFlow.collectLatest { pagingData ->
    pagingAdapter.submitData(pagingData)
  }
}

Paged data transformations

Displaying a filtered list

Transforming PagingData streams is very similar to the way you would any other data stream. For example, if we only want to display playful doggos from our Flow<PagingData<Dog>> we would need to map the Flow object and then filter the PagingData:

doggosPagingFlow.map { pagingData ->
        pagingData.filter { dog -> dog.isPlayful }
    }

List with separators

Adding list separators is also a paged data transformation where we transform the PagingData to insert separator objects into the list. For example, we can insert letter separators for our doggos’ names. When using separators, you will need to implement your own UI model class that supports the new separator items. To modify your PagingData to add separators, you will use the insertSeparators transformation:

pager.flow.map { pagingData: PagingData<Dog> ->
  pagingData.map { doggo ->
    // Convert items in stream to UiModel.DogModel.
    UiModel.DogModel(doggo)
  }
  .insertSeparators<UiModel.DogModel, UiModel> { before: Dog, after: Dog ->
      return if(after == null) {
          // we're at the end of the list
          null
      } else if (before == null || before.breed != after.breed) {
          // breed above and below different, show separator
          UiModel.SeparatorItem(after.breed)
      } else {
          // no separator
          null
      }
    }
  }
}.cachedIn(viewModelScope)

Just as before, we're using cachedIn right before the UI layer. This ensures that loaded data and the results of any transformations can be cached and reused after a configuration change.

Advanced Paging work with RemoteMediator

If you’re paging data from a layered source, you should implement a RemoteMediator. For example, in the implementation of this class you need to request data from the network and save it in the database. The load() method will be triggered whenever there is no more data in the database to be displayed. Based on the PagingState and the LoadType we can construct the next page request.

It’s your responsibility to define how the previous and next remote page keys are constructed and retained as the Paging library doesn’t know what your API looks like. For example, you can associate remote keys to every item you receive from the network and save them in the database.

override suspend fun load(loadType: LoadType, state: PagingState<Int, Dog>): MediatorResult {

   val page = ... // computed based on loadType and state

   try {
       val doggos = backend.getDoggos(page)
       doggosDatabase.doggosDao().insertAll(doggos)
       
       val endOfPaginationReached = emails.isEmpty()
       return MediatorResult.Success(endOfPaginationReached = endOfPaginationReached)
   } catch (exception: Exception) {
       return MediatorResult.Error(exception)
   } 
}

When you’re loading data from the network and saving it to the database, the database is the source of truth for the data displayed on the screen. This means that the UI will be displaying data from your database, so you’ll have to implement a PagingSource for your database. If you’re using Room, you’ll just need to add a new query in your DAO that returns a PagingSource:

@Query("SELECT * FROM doggos")
fun getDoggos(): PagingSource<Int, Dog>

The Pager implementation changes slightly in this case, as you need to pass the RemoteMediator instance as well:

val pagingSourceFactory = { database.doggosDao().getDoggos() }

return Pager(
     config = PagingConfig(pageSize = NETWORK_PAGE_SIZE),
     remoteMediator = DoggosRemoteMediator(service, database),
     pagingSourceFactory = pagingSourceFactory
).flow

Check out the docs to find out more about working with RemoteMediator. For a complete implementation of RemoteMediator in an app, check out step 15 of the Paging codelab and the accompanying code.

We’ve designed the Paging 3 library to help you accommodate both simple and complex uses of Paging. It makes it easier to work with large sets of data whether it’s being loaded from the network, a database, in-memory cache, or some combination of these sources. The library is built on coroutines and Flow, making it easy to call suspend functions and work with streams of data.

To get started, find out more about Paging in our documentation and try it out by taking our codelab or checking out the sample. Then, let us know how we can improve the library by creating issues on the Issue Tracker.