The latest Android and Google Play news for app and game developers.
Posted by Don Turner, Developer Advocate, Android Audio Framework
This week we released the first production-ready version of Oboe - a C++ library for building real-time audio apps. Oboe provides the lowest possible audio latency across the widest range of Android devices, as well as several other benefits.
Oboe takes advantage of the improved performance and features of AAudio on Oreo MR1 (API 27+) whilst maintaining backward compatibility (using OpenSL ES) on API 16+. It's kind of like AndroidX for native audio.
Using Oboe you can create an audio stream in just 3 lines of code (vs 50+ lines in OpenSL ES):
AudioStreamBuilder builder; AudioStream *stream = nullptr; Result result = builder.openStream(&stream);
Take a look at the short video introduction:
Check out the documentation, code samples and API reference. There's even a codelab which shows you how to build a rhythm-based game.
If you have any issues, please file them here, we'd love to hear how you get on.
Posted by Sami Tolvanen, Staff Software Engineer, Android Security
Android's security model is enforced by the Linux kernel, which makes it a tempting target for attackers. We have put a lot of effort into hardening the kernel in previous Android releases and in Android 9, we continued this work by focusing on compiler-based security mitigations against code reuse attacks.
Google's Pixel 3 will be the first Android device to ship with LLVM's forward-edge Control Flow Integrity (CFI) enforcement in the kernel, and we have made CFI support available in Android kernel versions 4.9 and 4.14. This post describes how kernel CFI works and provides solutions to the most common issues developers might run into when enabling the feature.
A common method of exploiting the kernel is using a bug to overwrite a function pointer stored in memory, such as a stored callback pointer or a return address that had been pushed to the stack. This allows an attacker to execute arbitrary parts of the kernel code to complete their exploit, even if they cannot inject executable code of their own. This method of gaining code execution is particularly popular with the kernel because of the huge number of function pointers it uses, and the existing memory protections that make code injection more challenging.
CFI attempts to mitigate these attacks by adding additional checks to confirm that the kernel's control flow stays within a precomputed graph. This doesn't prevent an attacker from changing a function pointer if a bug provides write access to one, but it significantly restricts the valid call targets, which makes exploiting such a bug more difficult in practice.
Figure 1. In an Android device kernel, LLVM's CFI limits 55% of indirect calls to at most 5 possible targets and 80% to at most 20 targets.
In order to determine all valid call targets for each indirect branch, the compiler needs to see all of the kernel code at once. Traditionally, compilers work on a single compilation unit (source file) at a time and leave merging the object files to the linker. LLVM's solution to CFI is to require the use of LTO, where the compiler produces LLVM-specific bitcode for all C compilation units, and an LTO-aware linker uses the LLVM back-end to combine the bitcode and compile it into native code.
Figure 2. A simplified overview of how LTO works in the kernel. All LLVM bitcode is combined, optimized, and generated into native code at link time.
Linux has used the GNU toolchain for assembling, compiling, and linking the kernel for decades. While we continue to use the GNU assembler for stand-alone assembly code, LTO requires us to switch to LLVM's integrated assembler for inline assembly, and either GNU gold or LLVM's own lld as the linker. Switching to a relatively untested toolchain on a huge software project will lead to compatibility issues, which we have addressed in our arm64 LTO patch sets for kernel versions 4.9 and 4.14.
In addition to making CFI possible, LTO also produces faster code due to global optimizations. However, additional optimizations often result in a larger binary size, which may be undesirable on devices with very limited resources. Disabling LTO-specific optimizations, such as global inlining and loop unrolling, can reduce binary size by sacrificing some of the performance gains. When using GNU gold, the aforementioned optimizations can be disabled with the following additions to LDFLAGS:
LDFLAGS += -plugin-opt=-inline-threshold=0 \ -plugin-opt=-unroll-threshold=0
Note that flags to disable individual optimizations are not part of the stable LLVM interface and may change in future compiler versions.
LLVM's CFI implementation adds a check before each indirect branch to confirm that the target address points to a valid function with a correct signature. This prevents an indirect branch from jumping to an arbitrary code location and even limits the functions that can be called. As C compilers do not enforce similar restrictions on indirect branches, there were several CFI violations due to function type declaration mismatches even in the core kernel that we have addressed in our CFI patch sets for kernels 4.9 and 4.14.
Kernel modules add another complication to CFI, as they are loaded at runtime and can be compiled independently from the rest of the kernel. In order to support loadable modules, we have implemented LLVM's cross-DSO CFI support in the kernel, including a CFI shadow that speeds up cross-module look-ups. When compiled with cross-DSO support, each kernel module contains information about valid local branch targets, and the kernel looks up information from the correct module based on the target address and the modules' memory layout.
Figure 3. An example of a cross-DSO CFI check injected into an arm64 kernel. Type information is passed in X0 and the target address to validate in X1.
CFI checks naturally add some overhead to indirect branches, but due to more aggressive optimizations, our tests show that the impact is minimal, and overall system performance even improved 1-2% in many cases.
CFI for arm64 requires clang version >= 5.0 and binutils >= 2.27. The kernel build system also assumes that the LLVMgold.so plug-in is available in LD_LIBRARY_PATH. Pre-built toolchain binaries for clang and binutils are available in AOSP, but upstream binaries can also be used.
The following kernel configuration options are needed to enable kernel CFI:
CONFIG_LTO_CLANG=y CONFIG_CFI_CLANG=y
Using CONFIG_CFI_PERMISSIVE=y may also prove helpful when debugging a CFI violation or during device bring-up. This option turns a violation into a warning instead of a kernel panic.
As mentioned in the previous section, the most common issue we ran into when enabling CFI on Pixel 3 were benign violations caused by function pointer type mismatches. When the kernel runs into such a violation, it prints out a runtime warning that contains the call stack at the time of the failure, and the call target that failed the CFI check. Changing the code to use a correct function pointer type fixes the issue. While we have fixed all known indirect branch type mismatches in the Android kernel, similar problems may be still found in device specific drivers, for example.
CFI failure (target: [<fffffff3e83d4d80>] my_target_function+0x0/0xd80): ------------[ cut here ]------------ kernel BUG at kernel/cfi.c:32! Internal error: Oops - BUG: 0 [#1] PREEMPT SMP … Call trace: … [<ffffff8752d00084>] handle_cfi_failure+0x20/0x28 [<ffffff8752d00268>] my_buggy_function+0x0/0x10 …
Figure 4. An example of a kernel panic caused by a CFI failure.
Another potential pitfall are address space conflicts, but this should be less common in driver code. LLVM's CFI checks only understand kernel virtual addresses and any code that runs at another exception level or makes an indirect call to a physical address will result in a CFI violation. These types of failures can be addressed by disabling CFI for a single function using the __nocfi attribute, or even disabling CFI for entire code files using the $(DISABLE_CFI) compiler flag in the Makefile.
static int __nocfi address_space_conflict() { void (*fn)(void); … /* branching to a physical address trips CFI w/o __nocfi */ fn = (void *)__pa_symbol(function_name); cpu_install_idmap(); fn(); cpu_uninstall_idmap(); … }
Figure 5. An example of fixing a CFI failure caused by an address space conflict.
Finally, like many hardening features, CFI can also be tripped by memory corruption errors that might otherwise result in random kernel crashes at a later time. These may be more difficult to debug, but memory debugging tools such as KASAN can help here.
We have implemented support for LLVM's CFI in Android kernels 4.9 and 4.14. Google's Pixel 3 will be the first Android device to ship with these protections, and we have made the feature available to all device vendors through the Android common kernel. If you are shipping a new arm64 device running Android 9, we strongly recommend enabling kernel CFI to help protect against kernel vulnerabilities.
LLVM's CFI protects indirect branches against attackers who manage to gain access to a function pointer stored in kernel memory. This makes a common method of exploiting the kernel more difficult. Our future work involves also protecting function return addresses from similar attacks using LLVM's Shadow Call Stack, which will be available in an upcoming compiler release.
Posted by Paul Bankhead, Director, Product Management, Google Play
We focus relentlessly on security and privacy on the Google Play Store to ensure Android users have a positive experience discovering and installing apps and games they love. We regularly update our Google Play Developer policies and today have introduced stronger controls and new policies to keep user data safe. Here are a few updates:
As previously announced, as of November 1, 2018, Google Play will require updates to existing apps to target API level 26 (Android 8.0) or higher (this is already required for all new apps). Our goal is to ensure all apps on Google Play are built using the latest APIs that are optimized for security and performance.
Our Google Play Developer policies are designed to provide a safe and secure experience for our users while also giving developers the tools they need to succeed. For example, we have always required developers to limit permission requests to only what is needed for their app to function and to be clear with users about what data they access.
As part of today's Google Play Developer Policy update, we're announcing changes related to SMS and Call Log permissions. Some Android apps ask for permission to access a user's phone (including call logs) and SMS data. Going forward, Google Play will limit which apps are allowed to ask for these permissions. Only an app that has been selected as a user's default app for making calls or text messages will be able to access call logs and SMS, respectively.
Please visit our Google Play Developer Policy Center and this Help Center article for detailed information on product alternatives to SMS and call logs permissions. For example, the SMS Retriever API enables you to perform SMS-based user verification and SMS Intent enables you to initiate an SMS or MMS text message to share content or invitations. We'll be working with our developer partners to give them appropriate time to adjust and update their apps, and will begin enforcement 90 days from this policy update.
In the coming months, we'll be rolling out additional controls and policies across our various products and platforms, and will continue to work with you, our developers, to help with the transition.
The trust of our users is critical and together we'll continue to build a safe and secure Android ecosystem.
Posted by James Lau (@jmslau), Product Manager
Today marks the beginning of KotlinConf 2018 - the largest in-person gathering of the Kotlin community annually. 2018 has been a big year for Kotlin, as the language continues to gain adoption and earn the love of developers. In fact, 27% of the top 1000 Android apps on Google Play already use Kotlin. More importantly, Android developers are loving the language with over 97% satisfaction in our most recent survey. It's no surprise that Kotlin was voted as the #2 most-loved language in the 2018 StackOverflow survey.
Google supports Kotlin as a first-class programming language for Android development. In the past 12 months, we have delivered a number of important improvements to the Kotlin developer experience. This includes the Kotlin-friendly SDK, Android KTX, new Lint checks and various Kotlin support improvements in Android Studio. We have also launched Kotlin support in our official documentation, new flagship samples in Kotlin, a new Kotlin Bootcamp Udacity course, #31DaysOfKotlin and other deep dive content. We are committed to continuing to improve the Kotlin developer experience.
As the language continues to advance, more developers are discovering the benefits of Kotlin across the globe. Recently, we traveled to India and worked with local developers like Zomato to better understand how adopting Kotlin has benefited their Android development. Zomato is a leading restaurant search & discovery service that operates in 24 countries, with over 150 million monthly users. Kotlin helped Zomato reduce the number of lines of code in their app significantly, and it has also helped them find important defects in their app at compile time. You can watch their Kotlin adoption story in the video below.
Android Developer Story: Zomato uses Kotlin to write safer, more concise code.
Going beyond Android, we are happy to announce that the Google Cloud Platform team is launching a dedicated Kotlin portal today. This will help developers more easily find resources related to Kotlin on Google Cloud. We want to make it as easy as possible for you to use Kotlin, whether it's on mobile or in the Cloud.
Google Cloud Platform's Kotlin Homepage
Adopting a new language is a major decision for most companies, and you need to be confident that the language you choose will have a bright future. That's why Google has joined forces with JetBrains and established the Kotlin Foundation. The Foundation will ensure that Kotlin continues to advance rapidly, remain free and stay open. You can learn more about the Kotlin Foundation here.
It's an exciting time to be a Kotlin developer. If you haven't tried Kotlin yet, we encourage you to join this growing global community. You can get started by visiting kotlinlang.org or the Android Developer Kotlin page.
Posted by Jamal Eason, Product Manager, Android
Today, Android Studio 3.2 is available for download. Android Studio 3.2 is the best way for app developers to cut into the latest Android 9 Pie release and build the new Android App bundle. Since announcing this update of Android Studio at Google I/O '18, we have refined and polished 20+ new features and focused our efforts on improving the quality for this stable release of Android Studio 3.2.
Every developer should use Android Studio 3.2 to transition to using an Android App Bundle, the new app publishing format. With very minimal work, you can generate an app bundle with Android Studio. Once you upload your app bundle to Google Play you can distribute smaller, optimized apps to your users. Early adopters have already seen between 11% - 64% in app size savings with app bundles over the legacy APK app size.
Another feature you do not want to miss is the Energy Profiler. This new profiler gives you a set of tools that will help you diagnose and improve the energy impact of your app. Better device battery life is one of the top most user requests, and with the Energy Profiler in Android Studio 3.2, you can do your part in improving device battery life by making sure your app is using the right amount of energy at the right time.
Lastly, you should also check out the new Android Emulator Snapshots feature. By using this feature, you can quickly take a snapshot of the current state of your emulator which includes the current state of the screen, apps, and settings. You can resume or boot into your emulator snapshot in under 2 seconds. For any app developer looking for super- fast boot times, or seeking to run tests in a predictable Android environment, Android Emulator Snapshots is a game changing feature for app development
On top of these major features, there are 20 new features plus many under-the-hood quality refinements in Android Studio 3.2. By using Android Studio 3.2, you can also develop for the latest technologies ranging from Android Jetpack, to the latest in Google Artificial Intelligence (AI) APIs with Android Slices.
Thank you to those who gave your early feedback on both the canary and beta releases. Your feedback helped us improve the quality and features in Android Studio 3.2. If you are ready for the next stable release, and want to use a new set of productivity features, Android Studio 3.2 is ready to download for you to get started.
Below is a full list of new features in Android Studio 3.2, organized by key developer flows.
Slices Provider Template
android.useAndroidX=true
gradle.properties
Build Android App Bundle
Android Emulator Snapshots
Energy Profiler
startMethodTracing(String tracePath)
stopMethodTracing()
To recap, the latest canary of Android Studio 3.2 includes these new major features:
Build
Optimize
Check out the release notes for more details.
Download the latest version of Android Studio 3.2 from the download page. If you are using a previous canary release of Android Studio, make sure you update to Android Studio Canary 14 or higher. If you want to maintain a stable version of Android Studio, you can run the stable release version and canary release versions of Android Studio at the same time. Learn more.
To use the mentioned Android Emulator features make sure you are running at least Android Emulator v28.0.7+ downloaded via the Android Studio SDK Manager.
We appreciate any feedback on things you like, and issues or features you would like to see. Please note, to maintain high product quality, a couple features (e.g. Navigation Editor) you saw in earlier release channels are not enabled by default in the stable release channel. If you find a bug or issue, feel free to file an issue. Connect with us -- the Android Studio development team ‐ on our Google+ page or on Twitter.
Posted by Jason Woloz and Mayank Jain, Android Security & Privacy Team
Our Android and Play security reward programs help us work with top researchers from around the world to improve Android ecosystem security every day. Thank you to all the amazing researchers who submitted vulnerability reports.
In the ASR program's third year, we received over 470 qualifying vulnerability reports from researchers and the average pay per researcher jumped by 23%. To date, the ASR program has rewarded researchers with over $3M, paying out roughly $1M per year.
Here are some of the highlights from the Android Security Rewards program's third year:
As part of our ongoing commitment to security we regularly update our programs and policies based on ecosystem feedback. We also updated our severity guidelines for evaluating the impact of reported security vulnerabilities against the Android platform.
In October 2017, we rolled out the Google Play Security Reward Program to encourage security research into popular Android apps available on Google Play. So far, researchers have reported over 30 vulnerabilities through the program, earning a combined bounty amount of over $100K.
If undetected, these vulnerabilities could have potentially led to elevation of privilege, access to sensitive data and remote code execution on devices.
In addition to rewarding for vulnerabilities, we continue to work with the broad and diverse Android ecosystem to protect users from issues reported through our program. We collaborate with manufacturers to ensure that these issues are fixed on their devices through monthly security updates. Over 250 device models have a majority of their deployed devices running a security update from the last 90 days. This table shows the models with a majority of deployed devices running a security update from the last three months:
Thank you to everyone internally and externally who helped make Android safer and stronger in the past year. Together, we made a huge investment in security research that helps Android users everywhere. If you want to get involved to make next year even better, check out our detailed program rules. For tips on how to submit complete reports, see Bug Hunter University.
Posted by Jingyu Shi, Partner Developer Advocate, Partner DevRel
This is the second in a series of blog posts in which outline strategies and guidance in Android with regard to power.
Notifications are a powerful channel you can use to keep your app's users connected and updated. Android provides Notification APIs to create and post notifications on the device, but quite often these notifications are triggered by external events and sent to your app from your app server.
In this blog post, we'll explain when and how to generate these remote notifications to provide timely updates to users and minimize battery drain.
We recommend using Firebase Cloud Messaging (FCM) to send remote notifications to Android devices. FCM is a free, cross-platform messaging solution that reliably delivers hundreds of billions of messages per day. It is primarily used to send remote notifications and to notify client applications that data is available to sync. If you still use Google Cloud Messaging (GCM) or the C2DM library , both of which are deprecated, it's time to upgrade to FCM!
There are two types of FCM messages you can choose from:
You can set the priority to either high or normal on the data messages. You can find out more about FCM messages and message handling in this blog post on Firebase Blog.
FCM is optimized to work with Android power management features. Using the appropriate message priority and type helps you reach your users in a timely manner, and also helps save their battery. Learn more about power management features in this blog post: "Moar Power in P and the future".
All of the notifications that you send should be well-structured and actionable, as well as provide timely and relevant information to your users. We recommend that you follow these notification guidelines, and avoid spamming your users. No one wants to be distracted by irrelevant or poorly-structured notifications. If your app behaves like this, your users may block the notifications or even uninstall your app.
The When not to use a notification section of the Material Design documentation for notifications highlights cases where you should not send your user a notification. For example, a common use case for a normal priority FCM Data Message is to tell the app when there's content ready for sync, which requires no user interaction. The sync should happen quietly in the background, with no need for a notification, and you can use the WorkManager1 or JobScheduler API to schedule the sync.
If you are sending remote notifications, you should always post the notification as soon as possible upon receiving the FCM message. Adding any additional network requests before posting a notification will lead to delayed notifications for some of your users. When not handled properly, the notifications might not be seen at all, see the "avoid background service" section below.
⚠️ Avoid adding any additional network requests before posting a notification
Also keep in mind that, depending on the state of the device, user actions, and app behavior, one or many power saving features could be restricting your app's background work. As a result, your app's jobs and alarms might be delayed, and its ability to access the network might be restricted.
For all of these reasons, to ensure timely delivery of the notification, you should always show the notification promptly when the FCM message is received, before any other work like network fetch or scheduling jobs.
To post a notification upon the receipt of an FCM message, you should include all the data needed for the notification in the FCM message payload.
The same applies to data sync--we recommend that your app send as much data as possible in the FCM payload and, if needed, load the remainder of the data when the app opens. On a well-performing network, there's a good chance that the data will be synced by the time the user opens the app so the spinner won't be shown to the user. If network connectivity is not good, a notification will be sent to the user with the content in the FCM payload to inform the user in a timely manner. The user can then open the app to load all the data.
You can also encrypt FCM messages end-to-end using libraries like Capillary. The image below shows a general flow of how to handle FCM messages.
As convenient as FCM message payload is, it comes with a 4KB maximum limit. If you need to send a rich notification with an image attachment, or you want to improve your user experience by keeping your app in sync with media content, you may need more than the 4KB payload limit. For this, we recommend using FCM messages in combination with the WorkManager 1 or JobScheduler API.
If you need to post a rich notification, we recommend posting the notification first, with some of the content in the FCM message. Then schedule a job to fetch the remainder of the content. Once the job is finished, update the notification if it is still active. For example, you can include a thumbnail or preview of the content in the FCM payload and post it in the notification first. Then schedule a job to fetch the rest of the media files. Be aware that if you've scheduled jobs from the FCM message handler, it is possible that when the user launches the app, the scheduled job won't have finished yet. You should handle this case gracefully.
In short, use the data in the FCM message payload to post a notification and keep your app content updated first. If you still need more data, then schedule jobs with APIs like WorkManager 1 or JobScheduler API.
One common pitfall is using a background service to fetch data in the FCM message handler, since background service will be stopped by the system per recent changes to Google Play Policy (Starting late 2018, Google Play will require a minimum target API level ).
Android 9 Pie will also impose background execution limits when battery saver is on. Starting a background service will lead to IllegalStateException from a normal priority FCM message. High priority messages do grant you a short whitelist window that allows you to start a background service. However, starting a background service with a network call will put the service at risk of getting terminated by the system, because the short execution window is only intended to be used for posting a notification.
You should avoid using background services but use WorkManager 1 or JobScheduler API instead to perform operations in the background.
Android 6 Marshmallow introduced Doze. FCM is optimized to work with Doze, and you can use high priority FCM messages to notify your users immediately. In Doze mode, normal priority messages are deferred to a maintenance window. This enables the system to save battery when a device is idle, but still ensure users receive time-critical notifications. Consider an instant messaging app that sends users messages from friends or incoming phone calls or a home monitoring app sends users alarm notifications. These are some of the acceptable examples where you can use high priority FCM messages.
In addition, Android 9 Pie introduced App Standby Buckets and App Restrictions.
The table below shows how various power-management features affect message delivery behaviors.
★ Note: Starting January 2019, App Restrictions (in Battery Setting) will include restrictions on FCM messages. You can find out if your app is in the restricted state with the isBackgroundRestricted API. Once your app is in the restricted state, no FCM messages will be delivered to the app at all. This will apply to both high and normal priority FCM messages and when app is in either foreground or background.
App Standby Buckets impose different levels of restrictions based on the app's standby bucket. Based on which bucket your app belongs to, there might be a cap for the number of high priority messages you are allowed to send per day. Once you reach the cap, any subsequent high priority messages will be downgraded to normal priority. See more details in the power management restrictions.
High priority FCM messages are designed to send remote notifications or trigger actions that involve user interactions. As long as you always use high priority messages for these purposes, your high priority messages will be delivered immediately and remote notifications will be displayed without delay. In addition, when a notification from a high priority message causes a user to open your app, the app gets promoted to the active bucket, which exempts it from FCM caps. The example below shows an instant messaging app moving to the active bucket after the user taps on a notification triggered by a high priority FCM message.
However, if you use high priority messages to send notifications to the blocked notification channels or tasks which do not involve user interactions, you will run the risk of wasting the high priority messages allocated in your app's bucket. Once reaching the cap, you won't be able to send urgent notifications anymore.
In summary, you should only use high priority FCM messages to deliver immediate, time-critical notifications to users. Doing so will ensure these messages and subsequent high priority messages reach your users without getting downgraded. You should use normal priority messages to trigger events that do not require immediate execution, such as a notification that is not time-sensitive or a data sync in the background.
We highly recommend that you test your apps under all of the power management features mentioned above. To learn more about handling FCM messages on Android in your code, visit the Firebase blog.
Thank you for helping move the ecosystem forward, making better Android apps, and saving users' batteries!
Acknowledgements: This blog posts is in joint collaboration with FCM and Android teams.
1 WorkManager is the recommended solution for background processing once it's stable.
Play Console
