The latest Android and Google Play news for app and game developers.
Posted by Mo Yu, Android Security & Privacy Team
In a previous blog post, we talked about using machine learning to combat Potentially Harmful Applications (PHAs). This blog post covers how Google uses machine learning techniques to detect and classify PHAs. We'll discuss the challenges in the PHA detection space, including the scale of data, the correct identification of PHA behaviors, and the evolution of PHA families. Next, we will introduce two of the datasets that make the training and implementation of machine learning models possible, such as app analysis data and Google Play data. Finally, we will present some of the approaches we use, including logistic regression and deep neural networks.
Detecting PHAs is challenging and requires a lot of resources. Our security experts need to understand how apps interact with the system and the user, analyze complex signals to find PHA behavior, and evolve their tactics to stay ahead of PHA authors. Every day, Google Play Protect (GPP) analyzes over half a million apps, which makes a lot of new data for our security experts to process.
Leveraging machine learning helps us detect PHAs faster and at a larger scale. We can detect more PHAs just by adding additional computing resources. In many cases, machine learning can find PHA signals in the training data without human intervention. Sometimes, those signals are different than signals found by security experts. Machine learning can take better advantage of this data, and discover hidden relationships between signals more effectively.
There are two major parts of Google Play Protect's machine learning protections: the data and the machine learning models.
The quality and quantity of the data used to create a model are crucial to the success of the system. For the purpose of PHA detection and classification, our system mainly uses two anonymous data sources: data from analyzing apps and data from how users experience apps.
Google Play Protect analyzes every app that it can find on the internet. We created a dataset by decomposing each app's APK and extracting PHA signals with deep analysis. We execute various processes on each app to find particular features and behaviors that are relevant to the PHA categories in scope (for example, SMS fraud, phishing, privilege escalation). Static analysis examines the different resources inside an APK file while dynamic analysis checks the behavior of the app when it's actually running. These two approaches complement each other. For example, dynamic analysis requires the execution of the app regardless of how obfuscated its code is (obfuscation hinders static analysis), and static analysis can help detect cloaking attempts in the code that may in practice bypass dynamic analysis-based detection. In the end, this analysis produces information about the app's characteristics, which serve as a fundamental data source for machine learning algorithms.
In addition to analyzing each app, we also try to understand how users perceive that app. User feedback (such as the number of installs, uninstalls, user ratings, and comments) collected from Google Play can help us identify problematic apps. Similarly, information about the developer (such as the certificates they use and their history of published apps) contribute valuable knowledge that can be used to identify PHAs. All these metrics are generated when developers submit a new app (or new version of an app) and by millions of Google Play users every day. This information helps us to understand the quality, behavior, and purpose of an app so that we can identify new PHA behaviors or identify similar apps.
In general, our data sources yield raw signals, which then need to be transformed into machine learning features for use by our algorithms. Some signals, such as the permissions that an app requests, have a clear semantic meaning and can be directly used. In other cases, we need to engineer our data to make new, more powerful features. For example, we can aggregate the ratings of all apps that a particular developer owns, so we can calculate a rating per developer and use it to validate future apps. We also employ several techniques to focus in on interesting data.To create compact representations for sparse data, we use embedding. To help streamline the data to make it more useful to models, we use feature selection. Depending on the target, feature selection helps us keep the most relevant signals and remove irrelevant ones.
By combining our different datasets and investing in feature engineering and feature selection, we improve the quality of the data that can be fed to various types of machine learning models.
Building a good machine learning model is like building a skyscraper: quality materials are important, but a great design is also essential. Like the materials in a skyscraper, good datasets and features are important to machine learning, but a great algorithm is essential to identify PHA behaviors effectively and efficiently.
We train models to identify PHAs that belong to a specific category, such as SMS-fraud or phishing. Such categories are quite broad and contain a large number of samples given the number of PHA families that fit the definition. Alternatively, we also have models focusing on a much smaller scale, such as a family, which is composed of a group of apps that are part of the same PHA campaign and that share similar source code and behaviors. On the one hand, having a single model to tackle an entire PHA category may be attractive in terms of simplicity but precision may be an issue as the model will have to generalize the behaviors of a large number of PHAs believed to have something in common. On the other hand, developing multiple PHA models may require additional engineering efforts, but may result in better precision at the cost of reduced scope.
We use a variety of modeling techniques to modify our machine learning approach, including supervised and unsupervised ones.
One supervised technique we use is logistic regression, which has been widely adopted in the industry. These models have a simple structure and can be trained quickly. Logistic regression models can be analyzed to understand the importance of the different PHA and app features they are built with, allowing us to improve our feature engineering process. After a few cycles of training, evaluation, and improvement, we can launch the best models in production and monitor their performance.
For more complex cases, we employ deep learning. Compared to logistic regression, deep learning is good at capturing complicated interactions between different features and extracting hidden patterns. The millions of apps in Google Play provide a rich dataset, which is advantageous to deep learning.
In addition to our targeted feature engineering efforts, we experiment with many aspects of deep neural networks. For example, a deep neural network can have multiple layers and each layer has several neurons to process signals. We can experiment with the number of layers and neurons per layer to change model behaviors.
We also adopt unsupervised machine learning methods. Many PHAs use similar abuse techniques and tricks, so they look almost identical to each other. An unsupervised approach helps define clusters of apps that look or behave similarly, which allows us to mitigate and identify PHAs more effectively. We can automate the process of categorizing that type of app if we are confident in the model or can request help from a human expert to validate what the model found.
PHAs are constantly evolving, so our models need constant updating and monitoring. In production, models are fed with data from recent apps, which help them stay relevant. However, new abuse techniques and behaviors need to be continuously detected and fed into our machine learning models to be able to catch new PHAs and stay on top of recent trends. This is a continuous cycle of model creation and updating that also requires tuning to ensure that the precision and coverage of the system as a whole matches our detection goals.
As part of Google's AI-first strategy, our work leverages many machine learning resources across the company, such as tools and infrastructures developed by Google Brain and Google Research. In 2017, our machine learning models successfully detected 60.3% of PHAs identified by Google Play Protect, covering over 2 billion Android devices. We continue to research and invest in machine learning to scale and simplify the detection of PHAs in the Android ecosystem.
This work was developed in joint collaboration with Google Play Protect, Safe Browsing and Play Abuse teams with contributions from Andrew Ahn, Hrishikesh Aradhye, Daniel Bali, Hongji Bao, Yajie Hu, Arthur Kaiser, Elena Kovakina, Salvador Mandujano, Melinda Miller, Rahul Mishra, Damien Octeau, Sebastian Porst, Chuangang Ren, Monirul Sharif, Sri Somanchi, Sai Deep Tetali, Zhikun Wang, and Mo Yu.
Posted by Iliyan Malchev, Project Treble Architect
Last week at the 2018 Android Dev Summit, we demonstrated the benefits of Project Treble by showing the same Generic System Image (GSI) running on devices from different OEMs. We highlighted the availability of GSI for Android 9 Pie that app developers can use to develop and test their apps with Android 9 on any Treble-compliant device.
Launched with Android Oreo in 2017, Project Treble has enabled OEMs and silicon vendors to develop and deploy Android updates faster than what was previously possible. Since then, we've been working with device manufacturers to define Vendor Interfaces (VINTF) and draw a clear separation between vendor and framework code on Android devices.
Going forward, all devices launching with Android 9 Pie or later will be Treble-compliant and take full advantage of the Treble architecture to deliver faster upgrades. Thanks to Treble, we expect to see more devices from OEMs running Android 9 Pie at the end of 2018 as compared to the number of devices that were running Android Oreo at the end of 2017.
The GSI is built from the latest available AOSP source code, including the latest bug fixes contributed by OEMs. Device manufacturers already use GSI to validate the implementation of the vendor interface on their devices, and Android app developers can now harness the power of the GSI to test their apps across different devices. With GSI, you can test your apps on a pure AOSP version of the latest Android dessert, including the latest features and behavior changes, on any Treble-compliant device that's unlocked for flashing.
We're continuing to work on making GSI even more accessible and useful for app developers. For example, the GSI could enable early access to future Android platform builds that you can run on a Treble-compliant Android 9 device, so you could start app development and validation before the AOSP release.
If you are interested in trying GSI today, check out the documentation for full instructions on how to build GSI yourself and flash it to your Treble-compliant device.
Posted by Leo Sei, Product Manager on Android
As you may have heard from the Android Dev Summit, we announced that we're expanding support in Android to include Foldables, in preparation for upcoming devices from hardware partners like Samsung.
Here are a set of recommendations and information to make sure your application provides a great user experience on this new form factor (you can also check out the Android Dev Summit dedicated session here)
On this new form factor, your application could be transitioned from one screen to another automatically (eg. when folding / unfolding a foldable phone).
During this transition, your app will receive a configuration change for the new layout (and possibly density in some cases).
To provide a great user experience when changing from one screen to the other, you want to make sure your app properly support runtime configuration change.
How to test: Emulators for various devices should become available soon (eg., Samsung will publish a folding / unfolding emulator apk later in Q4 which should work on Samsung Galaxy S4 tablets as well as the AOSP emulator in Android studio).
Today, when an app is in multi-window but not focused, it is on the OnPause state.
While we provide recommendations on how to support multi-window, we noticed a significant number of apps are not handling the onPause state according to those recommendations (video paused or stopped, instant messages not displayed etc).
To help developers provide the best user experience on multi-window with minimal effort, we're allowing device manufacturers to keep all apps resumed when in multi-windows in P.
To opt-in to this behavior in Android P, add the following meta-data in your app manifest:
<meta-data android:name="android.allow_multiple_resumed_activities" android:value="true" />
Note: With the next Android version we're looking into how to optimize compatibility for this behavior.
How to test: There are no device at the moment with this behavior but device manufacturers are working to update existing devices to allow developers to test. Stay tuned for more details from device manufacturers.
Beginning with Android 8.0 (API level 26), the platform offers enhanced support for multiple displays. If an activity supports multi-window mode and is running on a device with multiple displays, users can move the activity from one display to another. When an app launches an activity, the app can specify which display the activity should run on. See here for the full documentation
How to test: You can try it out by using the "Developer options > Simulate secondary displays" option. Keep in mind that those simulated display do not process inputs.
Today, at the Computer History Museum in Mountain View, CA, we kicked off the Android Dev Summit, taking a look back at the last 10 years of Android and then jumping into some important new features for Android developers. Here's a look at some of the things we shared!
As early as Android 1.6, Android and our partners have contemplated different screen sizes and densities, enabling the platform to power a broad category of form factors and new experiences like Android TV, Android Auto, Wear OS and even Android apps on Chromebooks. Phone screens are an area where Android partners set the bar, introducing "phablets" when phone screens were small. Fast forward to today, when a phablet is... just a phone, a standard size users have come to love.
Now we see a Android device makers creating a new category: Foldables. Taking advantage of new flexible display technology, the screen can literally bend and fold.
There are two variants broadly speaking: two-screen devices and one-screen devices. When folded, foldables look like phones, fitting in your pocket or purse. When unfolded, their defining feature is what we call screen continuity. For example, start a video with the folded smaller screen - and later you can sit down and unfold the device to get a larger tablet-sized screen for a beautiful, immersive experience. As you unfold, the app seamlessly transfers to the bigger screen without missing a beat. We're optimizing Android for this new form factor. And, making changes to help developers everywhere take advantage of the possibilities this creates for amazing new experiences, new ways to engage and delight your users. Tune in to the Foldables session at Dev Summit this week to learn more. Expect to see Foldables coming from several Android manufacturers, including one Samsung previewed today and plans to offer next year.
We made Kotlin a first class language on Android in 2017. This month we had over 118,000 new projects using Kotlin started in Android Studio - from those users who opt in to share metrics. That's a 10X increase from last year. It's become the fastest growing language in terms of growth of number of contributors on GitHub, and voted the #2 most loved language on Stack Overflow. In our surveys, the more developers use Kotlin, the higher their satisfaction.
Last week, JetBrains released the latest version of Kotlin, 1.3, which brings new language features, APIs, bug fixes, and performance improvements:
All of these new features of Kotlin 1.3 will be integrated into the Kotlin-specific APIs that we provide–a majority of which are through KTX extensions as part of Jetpack.
At Google I/O we announced Jetpack, the next generation of tools and Android APIs to accelerate Android application development. Jetpack builds on the foundations laid out by Support Library and Architecture. Already, 80% of top 1,000 apps and games are using one of the new Jetpack libraries in production.
This summer we moved AndroidX - Jetpack's evolution of the original Android Support Library - to public AOSP. This means you can see features and bug fixes implemented in real-time, and contribute to any of the AndroidX libraries. You can learn more about contributing here.
We've been working to get as much feedback and refinement as possible on two new Architecture Component libraries: Navigation and Work Manager, and we plan to move both to Beta this month. The Navigation Architecture Component offers a simplified way to implement Android's navigation principles in your application, using a single Activity. Plus, the new Navigation Editor in Android Studio creates and edits your navigation architecture. This eliminates navigation boilerplate, gives you atomic navigation operations, easier animated transitions and more. WorkManager makes it easy to perform background tasks in the most efficient manner, choosing the most appropriate solution based on the application state and device API level.
Navigation Editor
We're also excited to see Android Slices move to public Search experiments! At I/O this year we introduced Slices, a new way to bring users to your app. Slices are like a mini snippet of your app, where you can surface content and actions. You can book a flight, play a video, or call a ride. Slices is another example where we want to be open very early, but we want to take the time to get it right. We're moving into public EAP this month with Doist, Kayak and others. We'll run experiments surfacing Slices in Google search results. To learn more, there's also a session today at Dev Summit with more info and best practices.
Android Studio is our official IDE for Android development. We asked where do you spend the most time? When we gather data from Android Studio's opted-in users we see that build time are getting faster with every release, sometimes as fast as 20%, but we also see build time getting slower and slower over time. So, how can both things be true? We've been digging in hard to understand.
It turns out build is a pretty complicated ecosystem. Developer choices makes a huge difference. Our developers are using a very broad (and growing) combination of OSes, custom plug-ins, annotation processors, languages. All of these can significantly affect times. In one case, a plugin some users like to add was silently slowing build speeds by up to 45%. Learning this, we realized we need build profiling and analysis tools so you can easily understand what's slowing your build down. We're also investing more in our own plugins to accelerate performance to make sure we continue to improve the performance of core build.
Android Studio 3.3 launches beta 3 today. In coming releases expect to see a strong focus on quality and fundamentals: reducing the number of crashes and hangs, optimizing memory usage, and fixing user-impacting bugs. We also announced today that we're making Android Studio an officially supported IDE on Chrome OS early next year; learn more here.
App sizes have grown dramatically, up 5x since 2012. But larger apps have downsides: lower install conversion rates, lower update rates, and higher uninstalls. This is why we built the Android App Bundle, the new publishing format that serves only the code and resources a user needs to run your app on their specific device; on average apps see 35% size savings compared to a universal APK. The app bundle also saves you time and effort with each release since you don't need to use incomplete solutions like multi-APK. Android Studio 3.2 brought full IDE support of app bundles, and there are now thousands of app bundles in production totaling billions of installs, including Google's apps like YouTube, Google Maps, Google Photos, and Google News.
The app bundle now supports uncompressed native libraries; with no additional developer work needed, the app bundle now makes apps using native libraries an average of 8% smaller to download and 16% smaller on disk on M+ devices.
Once you switch to the app bundle you can also start modularizing your app. With dynamic feature modules, you can load any app functionality on demand instead of at install time. You don't need to keep big features that are only used once, on every single device forever; dynamic features can be installed and uninstalled dynamically when your app requests them.
We've heard that you'd like more controls to ensure that users are running the latest and greatest version of your app. To address this, we're launching an In-app Updates API. We're testing the API with early access partners and will be launching it to all developers soon.
You'll have two options with this API; the first is a full-screen experience for critical updates when you expect the user to wait for the update to be applied immediately. The second option is a flexible update, which means the user can keep using the app while the update is downloaded. You can completely customize the update flow so it feels like part of your app.
We're also making instant apps easier than ever to adopt. We recently made using web URLs optional, enabling you to take your existing play store deep link traffic and send users to your instant experience if it's available. Additionally, we've raised the instant app size limit to 10MB for the Try Now button on the Play Store and web banners to make it even easier to adopt.
In the Android Studio 3.3 beta, you can now build an instant-enabled app bundle. This means that you can now build and deploy both your Instant and installed experiences from a single Android Studio project, and include them in a single Android App Bundle. You only have to upload just ONE artifact for both instant and installed app.
As developers, your feedback has been critical in shaping these investment areas; you are part of how we work, from early ideas, to EAPs and canaries, Beta, and iterating after launch. We hope you join us for the next two days whether you're watching the 30+ sessions on the livestream, joining social, or with us in-person in Mountain View. From the team, a sincere thank you for all your thoughtful feedback and contributions. We hope you enjoy Android Dev Summit.
Posted by Leo Sei, Product Manager on Android Studio and R8
Android developers know that APK size is an important factor in user engagement. Code shrinking helps reduce the size of your APK by getting rid of unused code and resources as well as making your actual code take less space (also known as minification or obfuscation).
That's why we're investing in improving code shrinking to make it faster and more efficient. We're excited to announce that the next generation code shrinker, R8, is available for preview as part of Android Studio 3.3 beta.
R8 does all of shrinking, desugaring and dexing in one step. When comparing to the current code shrinking solution, Proguard, R8 shrinks the code faster while improving the output size.
The following data comes from benchmark on the Santa Tracker app, you can find the project with benchmark details on this GitHub repository.
R8 is available with Android Studio 3.3 beta and works with Proguard rules. To try it, set the following in your project's gradle.properties file:
android.enableR8=true
For the more adventurous, R8 also has full mode that is not directly compatible with Proguard. In order to try that out you can additionally set the following in your gradle.properties file:
android.enableR8.fullMode=true
This turns on more optimizations, that can further reduce app size. However, you might need a few extra keep rules to make it work.
We have tested R8 correctness and performances on a number of apps and the results are promising so we plan to switch R8 as the default shrinker in Android studio soon.
Please give R8 a try and we would love to hear your feedback. You can file a bug report using this link.
Posted by Matt Pearring, Associate Product Marketing Manager, Developer Marketing
In just a week, we'll be kicking off Android Dev Summit 2018, broadcasting live from the Computer History Museum in Mountain View, CA on November 7 and 8. We'll have two days of deep technical sessions from the Android engineering team, with over 30 sessions livestreamed. The app just went live; download it on Google Play and start planning.
With the app you can explore the conference schedule with details on keynotes, sessions, and lightning talks. You can also plan your summit experience by saving events to your personalized schedule. This year's app is also an Instant app, so you can try it out first before installing it!
Android Dev Summit app screenshots
If you can't join in person, you can always join us online — we'll be livestreaming all of the sessions on the Android Dev Summit website or app and making them available on YouTube throughout the conference so you can watch at your own pace. Plus, we will share updates directly from the Computer History Museum to our social channels, so be sure to follow along!
Posted by Ritesh Nayak M, Product Manager
Three years ago, we created Nearby Notifications as a way for Android users to discover apps and content based on what is nearby. Our goal was to bring relevant and engaging content to users - to provide useful information proactively. Developers have leveraged this technology to let users know about free wifi nearby, provide guides while in a museum, and list transit schedules at bus stops.
We've learned a lot building and launching Nearby Notifications. However, earlier this year, we noticed a significant increase in locally irrelevant and spammy notifications that were leading to a poor user experience. While filtering and tuning can help, in the end, we have a very high bar for the quality of content that we deliver to users, especially content that is delivered through notifications. Ultimately, we have determined these notifications did not meet that bar. As a result, we have decided to discontinue support for Nearby Notifications. We will stop serving Nearby Notifications on December 6th, 2018.
Android users will stop receiving Nearby Notifications.
On December 6th we will stop delivering both Eddystone and Physical Web beacon notifications. You will still continue to have access to the beacon dashboard and can deliver proximity based experiences similar to Nearby Notifications via your own apps using our Proximity Beacons API.
We have two related APIs, Nearby Messages and Connections, that are available for developers to build device-to-device connectivity experiences, and also have Fast Pair, for device discovery and pairing. We will continue to invest in these APIs and support products using these technologies.
We sincerely appreciate the efforts of the Android developer community in supporting and evolving Nearby technology and the feedback that has helped us improve. We look forward to continuing to deliver engaging proximity experiences to users and seeing what developers create within their apps with our APIs.
Play Console
