CANFnet is a neural network architecture that extracts high-resolution contact area and normal force information from visuotactile sensor images, trained using labeled data from conventional force/torque sensors rather than complex material modeling. The system works effectively with commercial DIGIT and GelSight Mini sensors for real-time force control tasks, offering a plug-and-play solution that can generalize across sensors of the same type.
Visuotactile sensors are gaining momentum in robotics because they provide high-resolution contact measurements at a fraction of the price of conventional force/torque sensors. It is, however, not straightforward to extract useful signals from their raw camera stream, which captures the deformation of an elastic surface upon contact. To utilize visuotactile sensors more effectively, powerful approaches are required, capable of extracting meaningful contact-related representations. This paper proposes a neural network architecture called CANFnet that provides a high-resolution pixelwise estimation of the contact area and normal force given the raw sensor images. The CANFnet is trained on a labeled experimental dataset collected using a conventional force/torque sensor, thereby circumventing material identification and complex modeling for label generation. We test CANFnet using GelSight Mini sensors and showcase its performance on real-time force control and marble rolling tasks. We are also able to report generalization of the CANFnets across different sensors of the same type. Thus, the trained CANFnet provides a plug-and-play solution for pixelwise contact area and normal force estimation for visuotactile sensors. The models, dataset, and additional information are open-source at https://sites.google.com/view/canfnet.
@inproceedings{funk_canfnet,author={Funk, Niklas and Müller, Paul-Otto and Belousov, Boris and Savchenko, Anton and Findeisen, Rolf and Peters, Jan},title={CANFnet: High-Resolution Pixelwise Contact Area and Normal Force Estimation for Visuotactile Sensors Using Neural Networks},booktitle={Embracing Contacts-Workshop at ICRA 2023, London, UK},year={2023},url={https://openreview.net/forum?id=dUO0QQw4FW},}
