How to Scrape Data From Android Apps?
Analyze App Network’s Traffic using apk-mitm as well as mitmproxy
In case, you need to acquire data, which your app consumes, then you require to see from where it’s coming. It means observing the network traffic. With Android, Google has put up some barriers to doing so. Let’s see how to connect some tools for getting the treasure that you want.
Besides data scraping, you might need to look at the network traffic of the app to observe which data it collects on you. We are pretty obsessed with the data our devices emit. Therefore, we need to understand what these things want to get from us and how.
Thanks to X-Byte Enterprise Crawling, you have got a very simple and cross-platform way of seeing the traffic for Android apps. We have used a tool named apk-mitm that helps you adjust the network safety profile data from the app’s apk files so that we could view the traffic now. We show you how to utilize that now.
Things We Need
Instructions
1. Create and Begin an Android Virtual Device: Within Android Developer Studio, you need to start an AVD Manager from the tools menu for creating your emulator. No special settings are there that you require to fear about, you can utilize any device as well as the Android version. We have chosen a Pixel 3 having an Android preview version (named “R”). When completed, we were at the screen, which looks like given below.
2. Get an APK file as well as Modify that using APK-MITM: Then, we require an apk file for running our emulator. We have just looked for the website where you could do that as well as came across the link https://apkgk.com. After that, we have found as well as downloaded a file we required. Once downloaded, utilizing apk-mitm is only one line.
3. Setting up the Proxy: Using mitmproxy tools, we favor using mitmweb, as we have never learned the station keyboard shortcuts. To begin this on Ubuntu, from a station, we navigate to where we downloaded these tools, as well as type “. /mitmweb”. When it gets started, you need to have the screen given below.
4. Set an AVD to Utilize the Proxy: With advanced settings for the AVD, go to settings, and then Proxy for setting the proxy. You need to set that as given below. Utilize 127.0.0.1 as a hostname, 8080 as a port number, and hit apply. You might need to reboot the AVD — and long-press the power button for doing so.
5. Installing the mitmproxy CA Certificate on AVD: In the android emulator, you need to open a browser as well as navigate to http://mitm.it. After that, click on the android icon for downloading the certificate. This won’t install inevitably, so you need to visit Settings -> Security -> Encryption & Credentials -> install a certificate -> CA certificate for installing the certificate that you have just downloaded. You’ll get a notification if you’ve done that right in the case, mitmproxy is within the Trusted Credentials.
6. Installing the App as well as Start Showing Traffic: Install an app by dragging a file that you have made in step 2 on the screen of the device emulator. Then open that and you will start observing some traffic within the mitmproxy screen. Just go through the JSON file that one is grasping from the AWS S3. We bet we could have a similar file with Python is required.
That’s it from us.
Hopefully, it helps in your projects as well as helps in avoiding some more invasive apps. You can send us your feedback in the comments. And if you want to know more, contact X-Byte Enterprise Crawling.
Originally published at https://www.xbyte.io.
