We propose a end-to-end learning pipeline for sythesizing phase-only 3D holograms in real-time.
MIT researchers have created an integrated design pipeline that enables a user with no specialized knowledge to quickly craft a customized 3D-printable robotic hand.
Beichen Li, a PhD student in CDFG, has been chosen as the recipient of the 2022 MIT EECS Ernst A. Guillemin Thesis Award in Artificial Intelligence and Decision Making for his master’s thesis entitled “Computational Discovery of Microstructured Composites with Optimized Trade-Off between Strength and Toughness,” supervised by Professor Wojciech Matusik.
We propose a rapid design and fabrication workflow for pneumatic actuators with integrated sensing using digital machine knitting.
CDFG members Yunsheng Tian and Yifei Li’s paper “JoinABLe: Learning Bottom-up Assembly of Parametric CAD Joints” has been accepted to CVPR 2022. In this paper, we propose a learning-based method that assembles parts together to form joints without the help of object class labels or human guidance, and a large dataset containing CAD assemblies with rich information.
Three papers authored by CDFG members have been accepted to ICLR 2022, and two have been selected for an oral presentation! Pingchuan Ma and Tao Du will present their work “RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation.” Minghao Guo and Beichen Li will also present their paper, “Data-Efficient Graph Grammar Learning for Molecular Generation”. The third accepted paper, authored by CDFG member Jie Xu, is titled “Accelerated Policy Learning with Parallel Differentiable Simulation”. Congratulations to all the paper authors!
CDFG UROPs Jagdeep Singh Bhatia, Holly Jackson, and PhD students Yunsheng Tian, Jie Xu’s paper Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robots has been accepted to NeurIPS 2021. In this paper, we propose Evolution Gym, the first large-scale benchmark for co-optimizing the design and control of soft robots.
CDFG member Michael Foshey, and Mina Konaković Luković’s paper Accelerated discovery of 3D printing materials using data-driven multiobjective optimization has been accepted to Science Advances. In this paper, we present how machine learning can aid the discovery of useful 3D printing material formulations.
We are pleased to announce that CDFG member and 3rd year graduate student Alyssa Luo has been awarded a MathWorks Engineering Fellowship for AY 21-22. Her current research interests involve working on tactile sensing and smart textiles. Congratulations to Alyssa on her fellowship.
CDFG member Liang Shi has been awarded a 2021 Facebook Fellowship in the area of AR/VR computer graphics. Additionally, he has just completed a Summer internship (5/28-8/20) at Facebook Reality Lab Display System Research group. Congratulations to Liang on his fellowship as well as involvement in this Summer internship.
Five papers authored by CDFG members have been conditionally accepted to SIGGRAPH 2021! We are excited to share these works, and hope you’ll stay tuned for their official releases over the coming months.
CDFG member Yiyue Luo, Michael Foshey, and Wan Shou’s paper Intelligent Carpet: Inferring 3D Human Pose from Tactile Signals has been accepted to CVPR 2021. In this paper, we present a 3D human pose estimation approach using vision as supersion and the pressure maps recorded by a large-scale tactile carpet as input. This work is in collobaration with Professor Antonio Torralba’s group.
CDFG member Jie Xu’s paper Multi-Objective Graph Heuristic Search for Terrestrial Robot Design has been accepted to ICRA 2021. In this paper, Jie and CDFG members Andrew Spielberg and Allan Zhao present efficient methods for co-designing rigid terrestrial robots’ control and morphology (including discrete topology) with multiple objectives. Jie will be presenting this work at ICRA 2021 in June.
CDFG member Tao Du’s paper Underwater Soft Robot Modeling and Control with Differentiable Simulation has been accepted to IEEE RA-L. In this work, Tao, Josie Hughes, and CDFG UROP student Sebastien Wah demonstrate the power of a differentiable simulator in the task of planning the motion and narrowing the reality gap of a real-world, starfish-like soft robot. The paper will be online in the incoming weeks.
The paper Learning Human-environment Interactions using Conformal Tactile Textiles has been published in Nature Electronics 2021. In this paper, we present a full-body tactile textile for the study of human activities. We capture diverse human–environment interactions and demonstrate the capability of our platform to classify humans’ sitting poses, motions and other interactions with the environment, recover dynamic whole-body poses, reveal environmental spatial information and discover biomechanical signatures. Pleaes check out the project webpage for more details.
CDFG members Liang Shi, Beichen Li, Changil Kim, Petr Kellnhofer’s paper Towards Real-time Photorealistic 3D Holography with Deep Neural Networks is published at Nature 2021. In their paper, the team presents the first deep-learning-based system for real-time photorealistic 3D hologram synthesis. The paper advances both physically-based methods and introduces the first large-scale 3D hologram dataset. Please check out this summary video for more details!
CDFG members Yiyue Luo and Kui Wu will present their technical paper KnitUI: Fabricating Interactive and Sensing Textiles with Machine Knitting at CHI 2021. In their paper, Yiyue and Kui present KnitUI, a novel, accessible machine-knitted user interface based on resistive pressure sensing. They demonstrate KnitUI as a portable, deformable, washable, and customizable interactive and sensing platform. It obtains diverse applications, including wearable user interfaces, tactile sensing wearables, and artificial robot skin. Pleaes check out the project webpage for more details, and feel free to reach out to Yiyue and Kui at CHI this year!
CDFG members will be presenting 4 new papers at SIGGRAPH Asia 2020, spanning areas such as robot design, material appearance, and fluidic device optimization. Tao Du, Kui Wu, and Andy Spielberg will present their work on Functional Optimization of Fluidic Devices with Differentiable Stokes Flow. On the robotics side, CDFG members Allan Zhao, Jie Xu, Mina Konaković Luković, and Andrew Spielberg will present RoboGrammar: Graph Grammar for Terrain-Optimized Robot Design. Lastly, we present two papers focused on material modelling and appearance. PhD students Liang Shi and Beichen Li tackle digital material synthesis in their paper MATch: Differentiable Material Graphs for Procedural Material Capture, while researchers Michal Piovarči and Michael Foshey investigate physical material properties with their paper, Towards Spatially Varying Gloss Reproduction for 3D Printing.
For more information about these projects, please visit the project webpages linked above. We look forward to seeing you at our upcoming talks!
Mina Konaković Luković and Yunsheng Tian’s paper Diversity-Guided Multi-Objective Bayesian Optimization With Batch Evaluations has been accepted to NeurIPS 2020. In this paper, we propose a novel multi-objective Bayesian optimization algorithm called DGEMO that is used to automate the process of discovering the Pareto-optimal solutions for multi-objective problems, while minimizing the number of performed evaluations. Our algorithm predominantly outperforms relevant state-of-the-art methods on all the benchmark problems.
Andrew Spielberg’s paper A Simple, Inexpensive, Wearable Glove with Hybrid Resistive‐Pressure Sensors for Computational Sensing, Proprioception, and Task Identification has been published in Advanced Intelligent Systems. In this paper, Andy and his collaborators from Professor Daniela Rus’ lab presented a fully soft, wearable glove which is capable of real‐time hand pose reconstruction, environment sensing, and task classification. The design is easy to fabricate using low cost, commercial off‐the‐shelf items in a manner that is amenable to automated manufacturing. The system can reconstruct user hand pose and identify sensory inputs such as holding force, object temperature, conductability, material stiffness, and user heart rate, all with high accuracy. Check out this article at Advanced Science News to see more exciting applications unlocked by this new technology!
Two papers on the application of multi-objective optimization techniques in supervised learning and reinforcement learning have been accepted to ICML 2020: Prediction-Guided Multi-Objective Reinforcement Learning for Continuous Robot Control from Jie Xu, Yunsheng Tian, and Pingchuan Ma proposed an efficient evolutionary learning algorithm to find the Pareto set approximation for continuous robot control problems, and Efficient Continuous Pareto Exploration in Multi-Task Learning from Pingchuan Ma and Tao Du presented a novel, efficient method that generates locally continuous Pareto sets and Pareto fronts for multi-task learning problems. Details about the two papers will be available soon.
The paper Physical Realization Of Elastic Cloaking With A Polar Material was recently accepted to Physical Review Letters (PRL). In this paper, Wan Shou and Beichen Li collaborated with researchers at University of Missouri and Dalian University of Technology to propose a novel elastic cloak by designing and fabricating a new class of polar materials with a distribution of body torque that exhibits asymmetric stresses. The work sets a precedent in the field of transformation elasticity and should find applications in mechanical stress shielding and stealth technologies. Check out their paper to learn more about this breakthrough in material science!
Dr. Petr Kellnhofer’s work “Gaze360: Physically Unconstrained Gaze Estimation in the Wild” was accepted to ICCV 2019. In this paper, we present Gaze360, a large-scale gaze-tracking dataset and method for robust 3D gaze estimation in unconstrained images. Our proposed 3D gaze model extends existing models to include temporal information and to directly output an estimate of gaze uncertainty. Finally, we demonstrate an application of our model for estimating customer attention in a supermarket setting. This paper was was in collaboration with Professor Antonio Torralba’s group and Toyota Research Institute.
Our very own Postdoctoral researcher Changil Kim is joining Facebook in Seattle as a research scientist in September, after spending two and a half wonderful years in our lab. During his time in our lab, Changil has actively contributed to many research projects, including motion magnification, replicating oil painting, speech to face, and video processing. Congratulations to Changil and wish him all the best in Seattle!
During the past week, Alex’s two recent papers on knitting Neural Inverse Knitting: From Images to Manufacturing Instructions (ICML 19) and Knitting Skeletons: Computer-Aided Design Tool for Shaping and Patterning of Knitted Garments (UIST 19) have received lots of media attention, including a recent interview from BBC. The story is also widely reported by TechCrunch, 7News Boston, Mashable, Fortune, VentureBeat, Engadget, ZDNet, Geek, Market Research Finance, Interesting Engineering, Fibre2Fashion, and MIT News.
Alex’s latest work Knitting Skeletons: A Computer-Aided Design Tool for Shaping and Patterning of Knitted Garments is officially accepted to UIST 2019. In this paper, Alex and Liane present a novel interactive system for simple garment composition and surface patterning. Both casual users and advanced users can benefit from their system. Check out these beautiful samples to explore the new possibility brought by their tool!
Jie’s SIGGRAPH paper Learning to Fly: Computational Controller Design for Hybrid UAVs with Reinforcement Learning has been covered by media outlets including Engadget, UASWeekly, Mashable, VentureBeat, SlashGear, Robotics Business Review, Eletronics360, UberGizmo, DroneLife, Unmanned Aerial, Science Daily, and AI News in the past week. If you are at SIGGRAPH, come to room 151 and check out Jie’s presentation today!
In our latest work published in Science Advances, our group presented an automated system that designs and 3-D prints complex robotic actuators which are optimized according to an enormous number of specifications. We demonstrate the system by fabricating actuators that show different black-and-white images at different angles. One of our actuators portrays a Vincent van Gogh portrait when laid flat and the famous Edvard Munch painting “The Scream” when tiled an angle. We also 3-D printed floating water lilies with petals equipped with arrays of actuators and hinges that fold up in response to magnetic fields run through conductive fluids.
Our latest work on human grasping “Learning the signatures of the human grasp using a scalable tactile glove” was published in Nature. In this paper, we use a scalable tactile glove and deep convolutional neural networks to show that sensors uniformly distributed over the hand can be used to identify individual objects, estimate their weight and explore the typical tactile patterns that emerge while grasping objects. Using a low-cost (about US$10) scalable tactile glove sensor array, we record a large-scale tactile dataset with 135,000 frames, each covering the full hand while interacting with 26 different objects. This set of interactions with different objects reveals the key correspondences between different regions of a human hand while it is manipulating objects. Insights from the tactile signatures of the human grasp—through the lens of an artificial analog of the natural mechanoreceptor network—can thus aid the future design of prosthetics, robot grasping tools, and human-robot interactions.
Our PhD student Jie Xu will present his latest work “Learning to Fly: Computational Controller Design for Hybrid UAVs with Reinforcement Learning” at SIGGRAPH 2019 this summer. In this paper, Jie proposed a novel neural network controller design for hybrid UAVs, an aerial robot that is challenging to control due to its complex aerodynamic effects. His method allows us to directly apply a controller trained in simulation to real hybrid UAV hardware without any modification. Checkout his website and the project page to find more!
Alex and Tae-Hyun’s latest paper on neural inverse knitting is accepted to ICML 2019. In this paper, they introduce the new problem of automatic machine instruction generation using a single image of the desired physical product. Check out the project page to learn more about the way they tackle the problem and see a dataset of real knitting samples!
After spending one and a half wonderful years in our lab, our very own Postdoctoral researcher, Tae-Hyun Oh, joined Faceboook AI Research, Cambridge, MA, which is located across the street. During his stay at MIT, Tae-Hyun has contributed to a wide range of research projects, including Motion Magnification, Soft Segmentation, Speech2Face, and Inverse Knitting. Congratulations to Tae-Hyun, and wish him the best of luck in the future!
Yuanming’s latest paper ChainQueen was accepted to ICRA 2019. In this paper, he presented a differentiable deformable body simulator that opens up lots of possibilities for efficient controller and geometry design algorithms in robotics. Check out this video to see what amazining work can be done using his simulator!
Liang Shi’s Siggraph Asia paper: Deep Multispectral Painting Reproduction via Multi-Layer, Custom-Ink Printing, has been covered by lots of media outlets including TechCrunch, Forbes 1, Forbes 2, Fast Company, Engadget, The Telegraph, WIRED Italy, Daily Mail, MIT News, 3D Printing Media Network, MIT Sloan Management Review, New Atlas, and many more! Please check out his website for paper and data, or come to his talk if you are also at Siggraph Asia this week!
Our group will present three papers at SIGGRAPH ASIA 2018 in Tokyo this December: Narrow-Band Topology Optimization on a Sparsely Populated Grid, InverseCSG: Automatic Conversion of 3D Models to CSG Trees, and Deep Multispectral Painting Reproduction via Multi-Layer, Custom-Ink Printing. Congratulations to Yuanming Hu, Tao Du, and Liang Shi!
Three papers from our group have been accepted to ECCV this year: Learning-Based Video Motion Magnification, A Dataset Of Flash And Ambient Illumination Pairs From The Crowd, and Learning To Zoom: A Saliency-Based Sampling Layer For Neural Networks. More details will be available soon!
Our PhD student Subramanian Sundaram successfully defended his PhD thesis “3D-printing Form and Function” this morning. His thesis committee members are Prof. Wojciech Matusik, Prof. Marc Baldo, and Prof. Vladimir Bulovic. Congratulations to Dr. Sundaram and wish him all the best in his future endeavors!
Adriana Schulz, who recently defended her PhD thesis in our group, will present her work Interactive Exploration of Design Trade-Offs at SIGGRAPH 2018. In this paper, Adriana and her coauthors presented a new method to interactively explore the Pareto front in the performance space of a design problem. This is joint work with Professor Eitan Grinspun at Columbia University and Professor Justin Solomon at MIT CSAIL.
Our postdoctoral researcher Petr Kellnhofer received this year’s Eurographics Annual Award for Best PhD Thesis for his strong technical contribution in perceptual modeling for stereoscopic 3D by running dedicated perceptual studies, deriving new perceptual models from his findings, and integrating these models in computer graphics algorithms. Congratulations!
Our Ph.D. student James Minor was recently awarded the NSERC (Natural Sciences and Engineering Research Council of Canada) postgraduate scholarship for his academic excellence, research ability, and leadership abilities. This prestigious and highly competitive scholarship will support James financially for the next two years. Congratulations to James!
AutoSaw, a robotic carpenter developed in Adriana Schulz’s latest ICRA paper Robot Assisted Carpentry for Mass Customization, has attracted a lot of media attention from MIT News, BBC News, New Atlas, cnet, Designboom, BGR, engadget, Popular Mechanics, The Verge, Business Insider Australia, Inverse, Science Daily, IEEE Spectrum, New Atlas, Electronics360, and many more!
After spending two wonderful years in our group as a Postdoctoral researcher, Melina Skouras is joining the IMAGINE team at Inria Grenoble as a tenured research scientist in December. Melina has helped many of us in various research projects with her knowledge and expertise. Congratulations to Melina on her next move in career!
Congratulations to Liang Shi on his newly accepted paper Near-eye Light Field Holographic Rendering with Spherical Waves for Wide Field of View Interactive 3D Computer Graphics at SIGGRAPH Asia this year! In this paper Liang and his colleagues at Nvidia present a light field-based CGH rendering pipeline allowing for reproduction of high-definition 3D scenes with continuous depth and support of intra-pupil view-dependent occlusion. Want to learn more? Please check out their project page and this cool video!
Our very own Desai Chen successfully defended his PhD thesis Multiscale Methods for Fabrication Design this afternoon. His thesis committee members are Professor Wojciech Matusik, Professor Justin Solomon, and Professor Markus J. Buehler. Congratulations to Dr. Chen, and wish him the best of luck for everything in the future!
Subramanian Sundaram’s recent work 3D-printed self-folding electronics was selected as Today’s spotlight on MIT front page this morning! In this paper, Subra and his colleagues built a prototype self-folding printable device that includes electrical leads and a polymer “pixel” that changes from transparent to opaque when a voltage is applied to it. Check out this video to learn more about this tiny and cute self-folding robot!
We are pleased to announce that Yuanming Hu is joining our group as a PhD student starting from September 2017. Yuanming obtained his Bachelor’s in Computer Science from Yao class at Tsinghua University. His research interests include physics-based simulation and rendering, as well as many other topics in computer graphics. Yuanming is also the author of Taichi, a popular open-source computer graphics software that implements more than 40 Siggraph/ToG papers. Welcome!
Bo Zhu’s paper Two-scale topology optimization with microstructures was featured at the front page of MIT News today! In this paper, Bo proposed a method that can efficiently genrate high-resolution 3D-printable structures with more than a trillion elements with just a regular computer.
Tae-Hyun Oh is joining our group as a new postdoctoral research starting from this summer. He received his PhD from KAIST where he worked with Prof. In So Kweon and Prof. Jinwoo Shin. He is a recipient of Microsoft Research Asia fellowship, gold prize of Samsung HumanTech thesis award, Qualcomm Innovation award, and top research achievement awards from KAIST. Welcome Tae-Hyun!
Cables provide a natural, biomimetic means of actuating robots, but figuring out how to route them to get the motion you want can be unintuitive. Our methods automate this process, and make it easy to design puppets, animatronics, and robots, for novices and experts alike. See the full paper at SCA ‘17.
Tired of spending time meant for your mechanical design work actually just waiting for simulations to finish?
Last Sunday our PhD student Andrew Spielberg presented a talk on “Co-Optimization of Robots and Mechanisms” at this year’s RSS workshop on minimality (minimality.mit.edu). In this talk, Andy presented recent work (ICRA ‘17) on the co-optimization of whole-body robot trajectories and design parameters from high-level task specifications. Their method requires minimal user domain knowledge, requiring only a coarse guess of the target robot configuration sequence and a parameterized robot topology as input. Further, this method considers the physical constraints afforded by the actuators, and selects the minimal actuators possible for task feasibility. Following this work, he also presented methods for improving the fabricability of such mechanisms using cable-driven actuation (SCA ‘17). Andy then presented work (IJRR ‘17) which makes this otherwise mathematically-intensive design process more intuitive for human users by presenting more intuitive, end-to-end CAD-like robot design software. Finally, Andy discussed some of the most important open problems facing the automated robot design community moving forward.
This year our group is going to present 6 papers at Siggraph 2017:
New post-doctoral associate Petr Kellnhofer joined our group in January from MPI Informatik and Saarland University. Petr visited our group in 2016 and this time he will use the research and knowledge garnered during his successful PhD thesis work to enhance all of our research efforts. Welcome back Petr!
We have 2 papers accepted to Siggraph Asia this year: Computational Multicopter Design presents an interactive system for computational design, optimization, and fabrication of multicopters, and Stochastic Structural Analysis For Context-Aware Design And Fabrication proposes failure probabilities as a semantically and mechanically meaningful measure of object fragility.
aeroMorph - Heat-sealing Inflatable Shape-change Materials for Interaction Design got Honorable Mention Awards at UIST 2016. Congratulations to Melina Skouras and her co-authors!
We have two new papers accepted to ACM Siggraph Asia 2015, one to UIST 2015, and one to ACM Multimedia 2015.
We have three papers accepted to ACM Siggraph 2015: “Data-driven finite elements for geometry and material design”, “Fab forms: customizable objects for fabrication with validity and geometry caching”, “MultiFab: a machine vision assisted platform for multi-material 3D printing.” We will be also presenting a talk on “Interactive Robogami: Data-Driven Design for 3D Print and Fold Robots with Ground Locomotion” and a course on “Computational Tools for 3D Printing”.
New post-doctoral associates Melina Skouras from ETH Zurich and Bo Zhu from Stanford University join our group. We also welcome Petr Kellnhofer from MPI Saarbrucken.
Our paper on “Approximating Free-form Geometry with Height Fields for Manufacturing” is accepted to Eurographics 2015
Our paper “Improving visual quality of view transitions in automultiscopic displays” is accepted to ACM Siggraph Asia 2014.
Our paper “Anahita: A System for 3D Video Streaming with Depth Customization” is accepted to ACM Multimedia 2014.
New graduate students: Subra Sundaram, Alexandre Kaspar, and Wei Zhao join CDFG.
Our group has presented four publications at Siggraph 2014:
Professor Matusik has given a talk at the MIT-Europe Conference. The conference this year focusses on advanced manufacturing and new urbanism.
Professor Matusik has delivered a keynote “From Virtual to Real” at Eurographics 2014.