Enhancing Real-Time Vision Systems: Integrating Dynamic Vision Sensors with Vision Transformers to Increase Computational Efficiency

Annabelle Yao; Oliver Su; Sumedha Kumar

doi:10.54254/2755-2721/107/20241399

Article Information

Received: 15 November 2024

Published: 26 November 2024

DOI: https://doi.org/10.54254/2755-2721/107/20241399

Article Type: Research Article

Cite this article

Yao,A.;Su,O.;Kumar,S. (2024). Enhancing Real-Time Vision Systems: Integrating Dynamic Vision Sensors with Vision Transformers to Increase Computational Efficiency. Applied and Computational Engineering,107,73-82.

Data availability

The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.

Disclaimer/Publisher's Note

The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of EWA Publishing and/or the editor(s). EWA Publishing and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

About this Volume

Volume Title: ACE Vol.107

Part of Series: Applied and Computational Engineering

ISSN: 2755-2721 (Print) / 2755-273X (Online)

Copyright & License

[1]“What Are the Disadvantages of a Vision Sensor? | IndMALL, ” Jun. 25, 2024. https://www.indmall.in/faq/what-are-the-disadvantages-of-a-vision-sensor/#:~:text=In%20factory%20settings%2C%20the%20movement (accessed Jul. 07, 2024).

[2]Photonics Media editors, “Dynamic Vision Sensors Detect, Differentiate Movement in Real Time, ” Photonics.com, Aug. 13, 2019. https://www.photonics.com/Articles/Dynamic_Vision_Sensors_Detect_Differentiate/a65017 (accessed Jul. 07, 2024).

[3]“Dynamic Vision Sensors – A Brief Overview - Image Sensor TechStream Blog | TechInsights, ” www.techinsights.com.https://www.techinsights.com/blog/image-sensor/dynamic-vision-sensors-brief-overview-image-sensor-techstream-blog#:~:text=Dynamic%20Vison%20Sensors%20are%20Asynchronous (accessed Jul. 07, 2024).

[4]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.

[5]A. M. George, D. Banerjee, S. Dey, A. Mukherjee and P. Balamurali, "A Reservoir-based Convolutional Spiking Neural Network for Gesture Recognition from DVS Input, " 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK, 2020, pp. 1-9, doi: 10.1109/IJCNN48605.2020.9206681.

[6]A. M. George, D. Banerjee, S. Dey, A. Mukherjee and P. Balamurali, "A Reservoir-based Convolutional Spiking Neural Network for Gesture Recognition from DVS Input, " 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK, 2020, pp. 1-9, doi: 10.1109/IJCNN48605.2020.9206681.

[7]W. He et al., Eds., “Comparing SNNs and RNNs on neuromorphic vision datasets: Similarities and differences, ” Neural Networks, Dec. 2020. https://www.sciencedirect.com/science/article/pii/S0893608020302902#sec5 (accessed Jul. 07, 2024).

[8]R. Graça, B. Mcreynolds, and T. Delbruck, “Shining light on the DVS pixel: A tutorial and discussion about biasing and optimization.” Accessed: Jul. 08, 2024. [Online]. Available: https://openaccess.thecvf.com/content/CVPR2023W/EventVision/papers/Graca_Shining_Light_on_the_DVS_Pix el_A_Tutorial_and_Discussion_CVPRW_2023_paper.pdf

[9]L. Meng et al., “AdaViT: Adaptive Vision Transformers for Efficient Image Recognition.” Accessed: Jul. 16, 2024. [Online]. Available: https://openaccess.thecvf.com/content/CVPR2022/papers/Meng_AdaViT_Adaptive_Vision_Transformers_for_Effi cient_Image_Recognition_CVPR_2022_paper.pdf

[10]Y. Kim and P. Panda, “Optimizing Deeper Spiking Neural Networks for Dynamic Vision Sensing, ” Neural Networks, vol. 144, pp. 686–698, Dec. 2021, doi: https://doi.org/10.1016/j.neunet.2021.09.022.

[11]O. Moutik et al., “Convolutional Neural Networks or Vision Transformers: Who Will Win the Race for Action Recognitions in Visual Data?, ” Sensors, vol. 23, no. 2, p. 734, Jan. 2023, doi: https://doi.org/10.3390/s23020734.

[12]M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy, “Do Vision Transformers See Like Convolutional Neural Networks?” Available: https://proceedings.neurips.cc/paper_files/paper/2021/file/652cf38361a209088302ba2b8b7f51e0-Paper.pdf

[13]“Getting started — 1.4.3, ” tonic.readthedocs.io. https://tonic.readthedocs.io/en/latest.

[14]PyTorch, “PyTorch documentation — PyTorch master documentation, ” Pytorch.org, 2019. https://pytorch.org/docs/stable/index.html

[15]Y. Kim and P. Panda, “Optimizing Deeper Spiking Neural Networks for Dynamic Vision Sensing, ” ScienceDirect, Dec. 2021. https://www.sciencedirect.com/science/article/pii/S0893608021003841

[16]“Papers with Code - DVS128 Gesture Dataset, ” paperswithcode.com. https://paperswithcode.com/dataset/dvs128-gesture-dataset

[17]"Event-based Vision Sensor(EVS)Technology | Technology, ” Sony Semiconductor Solutions Group. https://www.sony-semicon.com/en/technology/industry/evs.html

[18]"Event-based Vision Sensor(EVS)Technology | Technology, ” Sony Semiconductor Solutions Group. https://www.sony-semicon.com/en/technology/industry/evs.html

[19]https://github.com/anniey-17/Integrating_DVS_with_ViT_for_enhanced_real_time_processing_and_classification/tree/main

[20]“Accuracy Test definition | Psychology Glossary | AlleyDog.com, ” www.alleydog.com. https://www.alleydog.com/glossary/definition.php?term=Accuracy+Test#:~:text=An%20accuracy%20test%20is%2 0designed (accessed Jul. 08, 2024).

[21]neuralthreads, “Categorical cross-entropy loss — The most important loss function, ” Medium, Dec. 26, 2021. https://neuralthreads.medium.com/categorical-cross-entropy-loss-the-most-important-loss-function- d3792151d05b

[22]Z. Zhou et al., “SPIKFORMER: WHEN SPIKING NEURAL NETWORK MEETS TRANSFORMER.” Accessed: Jul. 25, 2024. [Online]. Available: https://arxiv.org/pdf/2209.15425v2

[23]G. Boesch, “Vision Transformers (ViT) in Image Recognition - 2024 Guide, ” viso.ai, Nov. 25, 2023.https://viso.ai/deep-learning/vision-transformer-vit/#:~:text=The%20visual%20transformer%20divides%20an