Apple

Software Engineering Intern @ Apple

May 2018 – December 2018

  • iOS Keyboards team member developing keyboard and autocorrect software for iPhone and iPad.

  • Developed the system-wide Intelligent Text Selection feature that shipped in iOS 13 to hundreds of millions of customers. Responsible for both frontend and backend logic. Presented the feature to Apple's SVP Software.

  • Researched machine learning techniques to detect QuickPath (swipe typing) inflection points. Developed an end-to-end pipeline including data visualization and labeling tools, implementation, and evaluation.

  • Conjured an algorithm to measure hand dimensions using iPad's capacitive touch sensor data. Calculates hand height, span, glove size, and handedness using a single "handprint".

 

Intelligent Text Selection

Double-tap to quickly select addresses, phone numbers, email addresses, and more.

I developed the Intelligent Text Selection feature including both the frontend (UIKit) and backend logic. It augments the built-in word tokenizer by automatically expending text selections to phone numbers, email addresses, named entities, etc. This means when you double-tap a phone number the entire entity is selected, not just the few numbers identified by the tokenizer. I had the honor of demoing the feature to Craig Federighi, Apple’s SVP Software, as well as other high-ranking Apple execs (they liked it!).

“Thank goodness there are still people @Apple that care about details like this.”
— Matt Sephton (@gingerbeardman) [1]
“Great quality of life feature.”
— Danny (@dancakebatter) [3]
“I love these little gems from Apple!”
— Anders Kjærsgaard (@ankjaers) [2]
“Thinking about the amount of work that went into this makes my head hurt.”
— Chris Wagner (@cwagdev) [4]

Also featured in:

  • Here are the best iOS 13 features you missed (The Next Web) [5]

  • Apple's Big iPad Update Reveals Something Huge About How it Sees the Future (Time) [6]

  • iOS 13 brings new gestures for cursor navigation, text selections, quick undo/redo, effortless cut/copy/paste & more (iDownload Blog) [7]

Source: Apple, iOS 13 Preview Features [8]

Source: Apple, iOS 13 Preview Features [8]

 

QuickPath Typing

Type by swiping from one letter to the next.

QuickPath was my team’s major focus for iOS 13 and I contributed research towards inflection point detection. In other words, classifying significant points along the path such as curves over keys the user intended to “type”, which are fed into our path-to-word algorithm. This research involved developing an end-to-end machine learning pipeline including a custom labeling and visualization framework, algorithm experimentation and tuning, and evaluation. Additionally, I took ownership of an in-house user studies tool near the end of its development to prepare it for a large-scale typing study.

Source: Apple, iOS 13 Preview [9]

“I personally love [QuickPath] for typing longer sentences on bigger screens... it’s welcome on the iPhone.”
— Marques Brownlee, Top 5 iOS 13 Features [10]
“As a fan of the stock keyboard but also a fan of swipe-to-type, this update made me very happy.”
— MacRumors, iOS 13 Hands-On: First Look of New Features! [12]
“[QuickPath] is very smooth... and overall we really like the experience.”
— AppleInsider, Top 6 Features of iOS 13 and iPadOS [11]
“The new QuickPath swipe keyboard in iOS 13 is a useful new addition for typing... Personally, I’ve found myself really enjoying the new QuickPath keyboard in iOS 13.”
— 9to5Mac, The new QuickPath swipe keyboard [13]

Also featured in:

  • iOS 13 - 70+ Best New Features & Changes! (Brandon Butch) [14]

  • 10 Major Changes in iOS 13! (Michael Billig) [15]

  • iOS 13 Best NEW Features & Changes (iDeviceHelp) [16]

 

Handprint

For Apple’s Weeks of Code (internal two-week hackathon), my colleagues Goutham Mani, James Magahern, and I developed Handprint, an app to quickly collect hand measurements during user studies. Simply place your hand on iPad’s screen. Each finger’s width is captured on a separate canvas to improve accuracy. Measurements can be viewed right in the app or transferred to another app by scanning the QR code encoding these data.

I invented the algorithm that that computes hand height, span, glove size, and handedness from the raw capacitive touch data. Notably, the algorithm works in any orientation. Additionally, I developed the QR code data transfer mechanism and designed the UI.

 
handprint-qr-code.gif