Problem: Estimating food calories manually is time consuming and often inaccurate because users must identify foods and estimate portion sizes themselves.

Solution: We developed an AI smart glasses system that uses YOLOv8 for food recognition and stereo vision for volume estimation, then calculates calories using nutrition data.

Impact: The system provides fast, hands free calorie estimation, making dietary tracking more convenient and demonstrating the potential of AI powered wearable health technology. Most importantly, this offers better depth estimation than AI calorie apps because it uses 2 cameras.

To address this problem, we developed a trace-driven memory scheduling simulator that compares different tensor-management policies, including LRU and a priority-aware scheduling policy. Our final Phase 12 priority policy preserves LRU-style activation reuse while adding conservative future-aware weight prefetching for upcoming compute layers. We evaluated the policy using MobileNetV2 layer traces, 3-layer window-based design space exploration, and selected buffer sizes such as 192 KB and 512 KB. The 192 KB case connects to a SCALE-Sim-like buffer configuration, while the 512 KB case demonstrates reduced DRAM read traffic for selected DepthwiseConv2d windows. We further validated selected DRAM-level behavior using Ramulator and DRAMPower.

The impact of this project is that it shows how lightweight scheduling decisions can improve inference latency and, in selected cases, reduce DRAM-device energy. The study provides a practical methodology for analyzing tensor reuse, prefetch opportunities, and memory-energy tradeoffs in edge AI accelerators.

Here's our team distribution:

• Yihan Zhao — YOLOv8n Model Training & Development

• Wending Zhu — UI Simulation & User Interface Design

• Advik Iyer — Project Idea Proposal

• Chih-Jung Hsu — Latency Optimization & Performance

UI Basic Workflows:

First, we upload left/right stereo images or use bundled demos. Then we set some parameters: B (baseline): Distance between cameras in mm (~60 mm for iPhone) and fx (focal length): Camera focal length in pixels (~2852 for iPhone 1× portrait). Then we run analysis. This model detects foods, estimate volume, and calculate calories. Finally we can see detection overlay, depth map, AR nutrition panel, and narration. Additionally, we use Deepseek AI Narration, which requires API key in .env . We add a function of text to speech function. It will automatically output voice for results.

YOLO Traning:

We train the YOLOv8n model on the Nutrition5k dataset, which covers a total of 13 distinct food classes and is pre-divided into training, validation and test subsets. We execute model training by running the command python train.py. Our core training hyperparameters are configured as follows: we select YOLOv8n, the nano-sized variant prioritizing fast inference speed; we set the total training epochs to 100, paired with an early stopping rule that terminates training if no performance gain is observed within 20 consecutive epochs. The batch size is fixed at 16, a value optimized for the NVIDIA A100 GPU, and we adopt a uniform input image resolution of 640×640. All training logs, weight files and result visualizations are automatically stored in the runs/ directory. In terms of recognition scope, this model is capable of identifying all 113 food categories from the Nutrition5k dataset, including a wide range of fruits and vegetables such as apples, bananas, bell peppers and avocados.

As a concrete methodology, a pedestrian crossing scenario was constructed within the open-source autonomous driving simulator "CARLA." Ranging experiments for each sensor were conducted under randomized adverse weather conditions, and time-series logs were collected and analyzed at a high sampling rate of 20 FPS (Delta t = 0.05 s). First, the baseline performance of each individual sensor was evaluated against the ground-truth (GT) distance under an ideal environment characterized by zero precipitation, zero fog, and a 45 dgree solar altitude angle (true noon). The results demonstrated that the stereo camera suffered from occasional ranging errors due to intense sunlight contrasts, as its recognition performance heavily relies on structural contrast. The AI camera exhibited a decline in accuracy at long ranges exceeding 20 meters, yet it demonstrated the highest tracking performance among the three sensors within a short range of 20 meters or less. Meanwhile, the semantic LiDAR proved to be completely distance-independent in the absence of weather noise, exhibiting exceptionally high stability, particularly at close ranges of 15 meters or less.

Next, to comprehensively evaluate the multi-modal impacts of meteorological variations and ambient lighting shifts, rainfall and fog parameters were normalized into 0.1 increments to visualize an "Ultimate 9-Matrix ODD Analysis" isolated across three distinct time slots: noon, sunset, and night. To ensure a stringent automotive safety standard, a successful recognition flag was defined as establishing a stable lock-on distance where the relative error against the ground-truth remained within 30% (alpha = 0.3) for 10 consecutive frames (0.50 seconds). The analysis revealed that during sunset, the low solar altitude angle causes intense backlighting (glare) to hit the camera lenses directly. This induces severe perception latency, forcing the vehicle to get approximately 10 meters closer to the target compared to noon before it can initiate a stable distance estimation. Furthermore, in nighttime environments under most adverse weather scenarios, camera-based modalities (stereo and AI) suffered from complete blindness until the target approached within a critical range of approximately 10 meters, highlighting a fatal recognition boundary.

In the final stage, using the $11 \times 11$ experimental matrix as a base dataset, raw empirical data points were 100% preserved while the remaining unmeasured sparse cells were smoothly interpolated based on local trends via grid-data interpolation (Griddata method). This synthesized a seamless, continuous contour map with a color range scaled from 5 meters (red: critical hazard) to 35 meters (blue: safe cruising). By overlaying a physical safety line of 18 meters—representing the total stopping distance (perception delay plus braking distance) when traveling at 40 km/h on a wet road surface with a friction coefficient of $\mu = 0.4$—the gap between perception limits and physical control constraint was evaluated. The quantitative results conclusively proved that regardless of the time of day (noon, sunset, or night), when the environment reaches extreme conditions of "fog density of 80% or higher" and "precipitation rate of 85–90% or higher," the stable tracking distance completely drops below this 18-meter threshold. Consequently, the vehicle is forced into a "braking limit violation zone" where a collision becomes physically unavoidable even at a moderate speed of 40 km/h. Ultimately, the outcomes of this research mathematically and physically visualize the hard boundaries of autonomous perception, delivering a solid engineering foundation for the safety-critical architecture design of all-weather autonomous driving systems.

Traditional static multi-sensor fusion algorithms fail to adapt to asymmetric performance drops of cameras, LiDAR and visual sensors under adverse weather. This project delivers an integrated simulation, algorithm and test framework to build all-weather reliable autonomous perception systems.

Main Functional Modules

1. Quantitative Sensor Performance Evaluation

   It quantifies performance degradation of stereo cameras, AI vision cameras and LiDAR under varied meteorological and lighting conditions, and identifies inherent detection limits such as LiDAR’s stable effective range and vision sensors’ fog interference thresholds.

2. Weather-Adaptive Dynamic Multi-Sensor Fusion

   Unlike static fusion strategies, the framework dynamically assigns weight to each sensor based on real-time weather and light input. It elevates the weight of weather-resistant LiDAR in fog or heavy rain and suppresses unreliable visual data under strong glare to mitigate perception failures.

3. Automated High-Speed Edge Case Mining

   The pipeline automatically generates rare high-risk driving scenarios for speeds up to 40 km/h and above. It discovers hidden safety hazards like insufficient braking distance triggered by sensor malfunction, enriching test coverage for extreme driving conditions.

4. End-to-End Automated CARLA Experiment & Data Operation System

• Deploys and maintains a fully automated simulation pipeline based on the CARLA simulator

• Implements an automatic data collection system integrated with Lark Bot and Git monitoring

• Supports segmented file upload, real-time exception alerting, and full workflow automation for massive repeated simulations

• Standardizes data generation, quality control and experimental result delivery

5. Weather Scenario Engineering & Hyperparameter Tuning

• Enables customized sequential weather variation scene design in simulation

• Embeds Optuna for automated hyperparameter search to optimize fusion model performance across weather profiles

Project Value

This toolkit resolves the poor generalization of conventional sensor fusion under complex weather. It combines adaptive perception algorithms with automated simulation testing, providing reproducible technical support for the development of safe, all-weather autonomous driving systems.

Our speaker was Yifei Li, a PhD graduate in Computer Science from MIT, where her research sits at the intersection of graphics, differentiable simulation, and physical AI. Before that, she completed her Bachelor's at Carnegie Mellon University — one of the top CS programs in the world.

Over the course of her academic career, she completed seven internships at companies including Activision Blizzard, Google, Meta Reality Lab, Boston Dynamics AI Institute, and NVIDIA. Each one took her somewhere different: game engines, augmented reality, digital humans, robotics. She wasn't chasing trends. She was following the problems.

This workshop was her telling us what she learned.

Chapter 1: The First Internship — Graphics Is Engineering

"Graphics means making things very fast to run in real time, writing code that other engineers can use, optimizing performance to its extreme."

When Yifei was a sophomore at Carnegie Mellon, she spent a summer writing code that made grass move.

Not metaphorically. Real grass — thousands of individually generated grass blades, swaying with wind, shifting in detail as the camera moved — all rendered in real time across the massive Battle Royale map of Call of Duty: Black Ops 4. It was her first internship, at Activision Blizzard. By the end of it, her work had shipped on every console and mobile platform the game touched.

Before that summer, she thought graphics was mostly about visual algorithms. After it, she understood something different: graphics in industry means performance, systems thinking, and code that survives a production environment. The grass looked beautiful. But she spent most of those months optimizing it.

Chapter 2: Google Maps — Making Virtual Objects Feel Real

"Small realism errors can break immersion. There are actually many things that could go wrong."

Her next stop was Google Maps, on the augmented reality navigation team. The challenge: when virtual information widgets float in front of real buildings on your phone screen, how do you make them look like they actually belong there?

The answer was the sun. By knowing a user's GPS coordinates and exact time, you can calculate the sun's position and render accurate shadows on virtual objects — shadows that match the real world around them. Yifei built that lighting estimation pipeline. It had to work in sunshine, in overcast weather, across different cities, at any time of day. The engineering problem wasn't the idea. It was the robustness.

Chapter 3: Meta Reality Lab — The Research Project That Went Viral

"At the beginning, this was framed as a research project. I never imagined it would actually be used."

Graduate school shifted Yifei's work from product-focused engineering to open-ended research. At Meta Reality Lab, she tackled an unlikely problem: teaching an AI to animate hand-drawn humanoid characters — the lopsided, expressive, anatomically approximate kind that children draw.

The system had to detect a skeleton in an image that looked nothing like a real body, reconstruct the character's geometry, and apply real motion sequences to it. Nobody expected it to become a product.

It did. Meta released it as a public web demo. Millions of people used it. Those users generated the data the team needed to keep improving the model. The lesson Yifei took: the distance between a research idea and something people love is sometimes shorter than you think.

Chapter 4: The Bigger Picture — Riding Industry Shifts

"The skills underneath stayed the same. What they were applied to kept changing."

One of the most valuable parts of Yifei's talk wasn't about any single project. It was the pattern across all of them.

She entered the industry in 2018, when games were the natural home for graphics talent. By 2020, the focus was shifting to AI. By 2021, every major tech company was building toward the metaverse — digital humans, virtual avatars, physics simulation fast enough to feel real. By 2024, the wave was robotics.

Yifei followed each current. Her later internships at Boston Dynamics AI Institute and NVIDIA put her inside the physical world: designing robotic gripper shapes in simulation, optimizing finger geometry for manipulation, working on simulation-supervised visual reasoning for humanoid robots.

The skills — geometry, simulation, differentiable physics, rendering — stayed constant. Their application just kept expanding.

Chapter 5: Why Graphics Belongs Everywhere Now

"Graphics tells us what structure should look like. That's exactly what modern AI needs."

By the end of the session, the picture was clear: computer graphics isn't a niche skill. It's a language for describing the physical world.

Geometry. Material properties. Light. Motion. Contact. Force. These are the building blocks graphics has always worked with — and they're exactly what modern AI systems need when they try to reason about the real world. Whether that's a robot learning to pick up an object, a simulation training a neural network, or a differentiable physics engine that lets you optimize a design by describing what you want it to do.

The problems worth working on, Yifei said, are the ones where you have to convert messy real-world input into something structured and controllable. That's what graphics has always done.

Chapter 6: Q&A — What the Interview Actually Looks Like

"For research roles, it's more about a match on the skills. They want to make sure you have the right skills for the problem."

The session closed with questions from attendees. The one that generated the most detail: how do engineering and research interviews actually differ?

Engineering interviews test coding, systems design, and behavioral fit. Research interviews are a different shape: less focused on distributed systems or databases, more focused on whether your specific background matches the specific problem the team is working on. Expect a presentation round where you walk through your own research in depth — they want to see how you think.

It's a small distinction that changes how you prepare, and how you choose what to apply for.

Join the GTC

AI+X GTC hosts sessions like this regularly — researchers, engineers, and builders sharing what the work actually looks like from the inside. It's free, open to all international students, and built around one idea: the earlier you understand what's happening at the frontier, the better positioned you are when you get there.

If you want to go deeper — BlendED's NVIDIA PBL puts students inside the exact space Yifei described: visual computing, simulation, and AI systems that reason about the physical world. Learn more about the NVIDIA PBL →

Yifei Li is a researcher at MIT working at the intersection of computer graphics, differentiable simulation, and physical AI. Her work spans cloth simulation, fluid dynamics, robotic manipulation, and digital humans.