Today, we are excited to share the story of Xingyu Lu (陆星宇), an Apple developer and a Swift Student Challenge Distinguished Winner.
Xingyu’s journey began with a simple question that came to him during a class: Why can’t Apple Pencil be used for gaming?
What started as a moment of curiosity turned into a deep exploration of human–computer interaction. He researched the possibilities of Apple Pencil and independently created Pencil Nib, an innovative app that explores pressure-based interactions through Apple Pencil. His work stood out among participants from around the world, earning him the title of Swift Student Challenge 25 Distinguished Winner, an honor awarded to only 50 developers globally. As part of this recognition, he was invited to Apple Park for a special WWDC event.
But the story did not end there. After the release of the iPhone 17 Pro, Xingyu noticed an unexpected coincidence: the official Chinese name of the new Cosmic Orange finish is “星宇橙”, which contains the exact same two Chinese characters as his own name, “星宇”. For Chinese speakers, these characters carry a meaning related to stars and the universe, making the connection feel even more special. Whether it is simply a coincidence or something more meaningful, this unexpected connection became a moment he wanted to share.
Now, let’s dive into the story written by Xingyu himself, and explore this unforgettable journey through his own words and experiences:
Credit: Written entirely by Xingyu Lu based on his own experiences, this article is published with his permission and reproduced without modification by the publisher.
It is now July 11, 2026, and 276 days have passed since the release of the iPhone 17 Pro.
During this nearly one-year period, I have often found myself involuntarily looking back on this matter. It happened so quietly that, for the public, it might have caused a little confusion, yet it was quickly overshadowed by countless other events. Perhaps it was merely a coincidence—but that coincidence seemed so precisely aligned that I could no longer tell what was truly happening.
For me, however, this was so intense and astonishing that I would often question whether it was real. Regardless of how one interprets it, this is something that deserves to be engraved in my life forever.
Now, moving from indifference, to doubt, to shock, I calmly type these words. I cannot predict what this story will bring after it is published—perhaps nobody will click on it; perhaps it will change the public’s perception of Apple Pencil and spatial computing; or perhaps nothing will happen at all.
I do not know how long this article will be. Maybe it will be very long; maybe it will not. But none of these words were generated by AI—because I truly, genuinely want to share this story with you in a serious and sincere way.
Whenever I tell my friends about it, they are always surprised and excited. I also believe it is a story that will leave a lasting impression on you.
You do have my words.
To truly tell this story well, I want to begin with an English class from two years ago.
A Crazy Idea During a Moment of Distraction
In June 2024, I was attending a general education course called “Western Culture,” which was designed for students preparing to pass an English proficiency test. It was a sleepy afternoon, and the class sounded rather boring (in fact, it is widely acknowledged that attending lectures is one of the least efficient ways to acquire knowledge). Although I tried to empathize with the teacher’s explanation of the winding journey of “Monet’s Sunset” among collectors, my attention was gradually drifting away.
Well, at least as a learner who is curious, ambitious, eager to explore a broader world, understand more knowledge, and make university life more colorful, I opened Goodnotes on my iPad.
Of course, you could also argue that I started daydreaming simply because the class was unbearably dull. However, I could not open TikTok or YouTube, because our English teacher was extremely strict (she could walk around the classroom twenty times during a single lecture almost like an iRobot). Therefore, I could only secretly open Goodnotes to do something else while making sure that, if the teacher came near, I could instantly switch back to my digital textbook.
I randomly drew two strokes with the Apple Pencil, just like most people would do:
“But unlike usual, I suddenly noticed a very silly question ZZZ :
“Why are the thicknesses of these two lines different? What causes them?”
“Oh, Pressure sensitivity. When I drew these two lines, I applied different amounts of pressure, so their thicknesses became different.”
“Why is it only now that I have truly noticed this? I understood the principle before, but why did I only become genuinely aware of it today, from the deepest level of consciousness?”
“Oh my god, what are you even talking about? As early as 110,000 years ago, our ancestors were already holding stones, applying pressure, and carving patterns onto bone fragments. Applying pressure on a surface, sliding across it, and producing delicate patterns with different widths—this is knowledge written into our human genes. Are you not human?”
“Indeed, you mentioned it: apply pressure – slide – obtain a pattern. But modern touch interaction ignores the first step, which is exactly why I find this so surprising…However, apply pressure – slide… apply pressure – slide… This seems like an extremely natural and powerful feedback mechanism, yet we have overlooked it.”
“HMMM now that you put it that way, you’re right. Let’s return to the Apple Pencil you are holding right now. This small device has helped many people since its invention, especially when you used it to review university physics last semester. You mentioned that the ‘pressure–sliding mechanism’ has been ignored. But the question is: what scenarios actually require this interaction method? Or rather, are there situations where only this mechanism can transmit the user’s pressure while accumulating it along a path? I don’t think so…?”
Oh, wait!”
Why has nobody thought about using the Apple Pencil to play games??
Yes, we have indeed underestimated pressure sensitivity—even the application scenarios of Apple Pencil itself. Most of its uses have been limited to productivity.
In real life, we have long been accustomed to controlling things through force (for example, when driving). However, in current interaction systems, this intuitive and natural method is surprisingly uncommon. Most forms of “force input” are still controlled through displacement—the amount something moves (such as moving a joystick). Because of this, a direction with enormous potential has been overlooked.
Yep，there were indeed early explorations of pressure-based interaction, but most of them focused on a single point, and eventually disappeared. As we discussed, “for 110,000 years, humans have been accustomed to applying pressure and sliding across surfaces to create patterns”—perhaps this explains why 3D Touch failed to become a subconscious form of interaction: A SINGLE POINT !
Apple Pencil changed this situation. It combines pressure sensitivity with gestures that have already been highly developed and enriched over time, allowing users to intuitively and precisely control objects on a plane—or even objects within a space composed of multiple planes.
It is not merely a pencil, nor is it only a productivity tool.
It is something like a… **controller? A pressure-sensitive controller for games, spatial experiences, and interaction
”?”
And when it comes to transmitting pressure and vectors in the most effective way for a controller interaction.(Just think about it: how many years have humans been using pens?)**
Although I later completed my presentation in that class and even received a score of 10, my mind was no longer in the classroom at all.
After class, I frantically searched on Google. When I was shocked to find there wasn’t much research on this, I made up my mind to invent an interactive mechanism to revamp the Apple Pencil…
When Pencil Is No Longer Just a Tool for Creation and Writing…
After several months of research and programming, I created Pencil Nib, the world’s first game controlled entirely through Apple Pencil interaction.
At the game start screen, players no longer “tap screen to start,” but instead double tap the Apple Pencil to enter the game.
The game contains four simple levels, corresponding to four different scenarios (drawing, connecting lines, maze, and pursuit). When the player hovers the Pencil over the cover of a level, the cover will float upward.
For Apple Pencil Pro, it will also trigger haptic in certain specific scenarios.
The most essential component in the game is a strange stick with a knob on it
(I know it looks a bit ugly)
In the game, players need to place the Pencil on the knob, and by applying different amounts of pressure, control the character’s speed, the color of the path (I evenly mapped seven colors to different pressure ranges), and the speed of the background music.
Additionally, users can make the knob slide up and down along the stick to control the direction of the character’s movement.
I’ve also deployed the pressure sensitivity mechanic into another roguelike I haven’t finished developing yet, called Moving Door, though this one uses a regular joystick layout.
Players still control how fast their tiny spaceship moves by varying how hard they press, guiding it over to the escape door scattered around the game levels.
But the escape door moves along with the tiny spaceship’s movement (I stacked several layers of triangle transformations and summed them up. )
If the escape door hits any edge of the screen, the game’s over. That means players have to carefully judge the ship’s speed and direction every single moment, and only the Apple Pencil can handle this simultaneous dual-parameter input.
Apple Pencil also has a greater mission, such as helping special groups of people achieve typing speeds closer to those of ordinary people.
A 2020 study published in The Lancet reported that 43.3 million people worldwide live with blindness, and this figure is projected to rise to 61 million by 2050.
Our world is largely built around sight and sound. Yet for people with deafblindness, a condition famously represented by Helen Keller, even the most basic text input remains a huge hurdle—voice typing is not an option for them.
That’s why I created an innovative specialized input method called Penboard. It bypasses standard keyboards and conventional touch interactions, both of which rely heavily on vision. Instead, it’s designed around two of humanity’s most intuitive physical instincts: applying pressure and sliding.
I’ve mapped every letter of the alphabet to distinct pressure ranges. When the user touches the screen with an Apple Pencil, a control knob pops up, and the corresponding letter selected via pressure will be highlighted on the alphabet panel to the left.
Once the highlighted letter matches the character they want to type, they slide the knob upward into the green zone to confirm input — the detected pressure value locks in place and won’t shift mid-slide. If they don’t drag the knob at all or lift the pencil off the tablet, no character gets entered, which effectively prevents accidental inputs. Sliding the knob down into the red zone deletes the last typed letter instead.
If I could embed a demo video right here, you’d be amazed to see that it takes less than half a second to select and confirm the letter “E” (or any other letter for that matter) purely through pressure input
I leveraged the native AVFoundation framework to deliver audio voice feedback every time a letter is selected or an action is performed. Need to type numbers or special symbols? Simply double-tap the Apple Pencil to switch between different character layouts. You can also tap or double-tap the screen with two finger.
The haptic vibration feature built into Apple Pencil Pro provides tactile feedback upon every letter selection, confirmation, and cancellation. This means users can complete text input relying solely on touch sensation, without any visual or auditory cues at all.
Lastly, the Apple Pencil shines remarkably in another unique use case, which also brings to light an overlooked issue none of us have really noticed before.
To keep you from getting bored reading this, grab any object you have nearby right now — literally anything. I’m sure you’re already holding something small as you read on. Now let’s break down what you just did (I’ll skip some redundant details to keep this concise, I know I’ve rambled quite a bit)🙃:
- Apply force – drag – grasp the object
- Pick up another item: Apply pressure to select – exert force to drag – grasp the object
- Do it again: Apply pressure to select – exert force to drag – grasp the object
- Once more! Apply pressure to select – exert force to drag – grasp the object
See the pattern here? In the physical world, we use force to single out objects and apply that same force to guide their movement until we finish our task.
Force exists everywhere in our 3D reality! Yet in VR environments, there’s no stable fulcrum for users to generate and exert physical force.
Isn’t it counterintuitive that we rely on all kinds of unnatural hand gestures to simulate the concept of force for 3D virtual spaces and feed those gestures into devices?
We interact with the physical world primarily through force, while we navigate 2D interfaces using taps and dragged paths made of discrete points. Force acts as the core interaction agent for 3D space, and simple points serve that role for 2D screens.
Spatial computing is currently lacking some expressive input methods built around force. When we move from flat 2D screens to immersive 3D environments, force input parameters become indispensable. Without them, no matter how flashy an interaction design looks, it’s nothing more than an artificial imitation of real-world physical manipulation.
That’s why I question the productivity potential of existing, in-development, and even conceptual VR/XR/MR human-computer interaction solutions. None of them capture genuine physical force. They lack robustness and make precise control and quantification extremely difficult.
(Sorry to skip ahead a few steps, but the logic is straightforward, I promise.)
Few people know that the Apple Pencil can detect its tilt angle relative to the iPad screen.
When you draw on an iPad with the Apple Pencil, you can visualize the pencil constantly slicing through the air above the display, like:
Taking this a step further: at every distinct tilt angle of the Apple Pencil, can we generate a normal plane anchored at the pencil’s tip — and enable all interactions to happen within that plane
In other words, we can project 3D space onto a 2D plane, then leverage pressure sensitivity within that plane for far more nuanced controls. This is exactly how we manipulate objects in the physical world, as a matter of fact.
This way of interacting—taking real-world pressure, directions and spatial positions and putting them into 3D space—I’ve come up with a brand new name for it.
I call it:
I believe the introduction of spatial pressure sensing unlocks the ability to precisely construct 3D models natively within spatial computing workflows. It allows fast, high-fidelity editing and remodelling of 3D assets, plus ultra-granular manipulation of all spatial elements inside immersive scenes.
(I’ve condensed the demo explanation as much as possible. Feel free to DM me or reach out via my email if you’d like to dive deeper into the technical details PLZ .)
These findings carry immense significance, yet my personal capacity is limited when it comes to fully researching and implementing this vision at scale.
I’m seeking to entrust this work to a larger, more resourceful organization capable of advancing it to its full potential.
I’ve decided to return all of this to its rightful home and hand the whole concept straight over to Apple.
The Highs and Heartaches Alike
I submitted all my visions for Apple Pencil applications spanning gaming, accessibility and spatial computing, alongside my demo, to the Swift Student Challenge 2025.
I waited for the results riddled with anxiety, honestly terrified my ideas might be too forward-thinking and get rejected outright. I even had a week-long stomach bug from all the stress leading up to the announcement
So you can imagine how thrilled and touched I was when Apple selected me as one of the world’s top 50 Distinguished Winners, reimbursed all my travel costs, and invited me to Apple Park for WWDC.
I also received an official certificate from Apple.
But then an unforeseen, completely unplanned disaster struck.
On June 3, 2025, at my third visa interview—after two prior rejections—the consulate took my certificate, event invitation, and travel itinerary away from me. A few days later, they even called to ask for my Apple Developer account login details and password. Turns out they’d been doubting whether my award certificate was genuine all along.
They did end up approving my visa in the end. Unfortunately, my flight had departed the day before.
I’d missed what was likely the one and only chance in my life to sit down face-to-face with Apple engineers and the Apple Pencil team to walk them through all my concepts. The three use cases I’d worked on held so much promise, yet I’d lost my opportunity to present them in person.
WWDC 2025 left me with an irreparable regret that I’ll carry forever.
For the next two months, I threw myself into hackathons and various competitions, hoping to distract myself from what had happened. No matter how hard I tried, though, every single morning when I woke up, my mind would drift back to that incident unbidden. I even kept beating myself up, wondering if things could’ve gone differently had I explained my project more thoroughly during my first two visa interviews.
I thought that was the end of the story. But what happened next far exceeded anything I could’ve imagined, bringing this whole journey to its real climax.
I think you can probably guess what came next.
“The Cosmic Orange”
September 9, 2025 marked Apple’s fall event—exactly 276 days ago from today.
Like countless other fans, I tuned in to watch the livestream. Ever since a GPS location glitch nearly made me miss my flight last time, I’d realized my iPhone 12 Pro was starting to fall behind the times.
Wow, the iPhone 17 looked awesome ! I loved that blue shade and the innovative front camera. Man, the Pro lineup got such massive upgrades this year, especially the new…
The new…
The new…
The new…WHAT ？？！
Wait wait wait wait, Why is my name featured on the new color variant of the iPhone 17 Pro ?
Yes, this is real; this actually happened. You can see from my certificate that my given name is written as “星宇”. It also means the universe. Below is a brief introduction to my name. Indeed, my name encompass continent, starry skies and the cosmos.
And the orange color of the iPhone 17 Pro is officially named Cosmic Orange in English. If you check Apple China’s official website, Wikipedia, or ask Siri or other AI agent for the official Chinese name of this orange shade, you’ll find it’s translated as ——
“星宇橙”
I’ve gradually calmed down from my excitement. Perhaps the lingering disappointment from WWDC left me subconsciously tangled with sorrow, quiet pride and detachment deep inside.
I’m just an ordinary person; all of this is nothing more than a sheer coincidence.
Still, after much thought, I ended up buying the orange iPhone 17 Pro.
Upgrading after a five-year gap brought a truly massive leap forward. The iPhone 17 Pro boasts an incredible display that retains exceptional color accuracy while remaining clearly visible even under direct sunlight. Its brand-new camera system has helped me capture countless precious memories—last winter, for instance, I took dozens of stunning snowscape photos when I was in Beijing.
Life went on uneventfully until one day.
“By the way, have you ever thought about the connection between ‘星宇’ and the orange ?”
A classmate casually brought this up over beef brisket rice while we were eating in the cafeteria.
“You really don’t find this suspicious? Doesn’t it feel way too coincidental?”
I was suddenly lost for words. But his question reminded me that I actually still owned another something orange-related.
Back in my dorm, I laid all the items out side by side. Something astonishing unfolded before my eyes:
I was utterly stunned.
Nevertheless, the reason I wrote this article to reveal the story behind all of this to you is not to boast that this year’s iPhone was named after me, nor to show off this one-of-a-kind story to flaunt myself.
No, far from it. Fate can be so dramatic and bizarre, yet I pen all these words with complete calm.
Absolutely not.
Epilogue
In summer, the damp sea breeze off the island is like a young girl, brushing the white-tiled teaching building with warm, moist fingertips. The aged pale tiles carry faint yellow stains blotted by the humid southern air.
Then she tiptoes and slips past half-opened windows: the scent of sweat, chalk dust, scraping chairs, and the rustle of pages being flipped all linger beneath the hushed excitement of students—their minds long since drifted past the playground, out to the sun-softened asphalt road beyond the school gate.
At last, she grows weary and pauses at the school entrance. Outside the turnstiles, parents wear anxious beads of sweat, yet barely conceal their inner excitement. It is Friday afternoon, the hour when my middle school lets out.
I lift my gaze from the computer screen back to the real world, looking out the floor-to-ceiling windows of the Starbucks:
Food delivery riders pull over to the curb, bowing their heads to confirm the destination of their next order; parents holding their children’s hands slow their pace, patiently answering their kids’ endless questions about the world.
Everyone strives to play their own part in their respective roles, striving to be a thread woven into society. Every trivial bit of effort eventually merges into the tide of the times, becoming force that propels society forward.
Every iteration of AI Agents, every breakthrough in algorithms, every newly born technology—technologies and products powerful enough to transform the world and enrich daily life—can never be embodied by a single individual. They are the crystallization of countless people’s dedication.
Therefore, what I wish to convey and clarify is this:
I wanna say congrats to the A-series chip team. The A19 Pro delivers staggering single-thread performance and exceptional task scheduling capabilities, while the upgraded Neural Engine unlocks greater potential for on-device large language models.
II wanna say congrats to the iPhone imaging team. The iPhone 17 Pro’s Pro Fusion system, paired with Center Stage, redefines self-portraiture and video calling. It enables vertical selfies with enhanced image quality, and Pro Fusion fuses footage from multiple focal lengths to deliver a far more consistent photographic experience.
I also wanna say congrats to the materials engineering team. The iPhone 17 Pro’s aerospace-grade 7000-series aluminum unibody, paired with its VC vapor chamber cooling system, delivers outstanding thermal dissipation efficiency.
I wanna say congrats to every friend currently or formerly employed at Apple. Each of you has contributed, in big or small ways, to bringing every one of Apple’s extraordinary products into existence.
I wish to extend my gratitude to Apple Developer Worldwide Relations, alongside countless other teams, for your courage in upholding “Think Different”.
I am thankful that Apple believes in the creative potential of the new generation of developers. Even amid the prevalence of vibe coding, you remain willing to give fledgling yet bold, passionate ideas a chance to be seen.
Last of all, I hold this belief:
A truly great form of interaction never hinges on how complex its underlying technology is. Instead, it hinges on how seamlessly it lets us overlook the complex technology powering it, while simplifying our lives.
We never stop to wonder why tapping an app icon on the home screen launches an application, nor why tapping different icons in the status bar navigates between pages.
We take these functions for granted. Only upon deeper reflection do we recognize the ingenious imagination and design prowess behind these interactions.
Perhaps one day in the future, we will no longer marvel at someone playing games with an Apple Pencil, nor grow curious upon seeing it used as an input device for spatial scenes—much like how we feel no confusion when gazing at symbol-carved stone tablets in museums.
You won’t merely grow to love it—no, you will grow fully familiar and accustomed to this mode of interaction. Deep down, every single one of us is still primal human at our core, merely trying on garments woven by the passage of time in life’s changing room.
When technology circles back to understanding humanity, when digital-age innovation integrates one of humanity’s oldest interaction mechanisms born 130,000 years ago for counting and record-keeping—the act of “apply pressure – slide”—we arrive at a brand-new answer to one of the biggest missing pieces in human-computer interaction: how to input the most primal variable behind sliding: force.。
🙂
