Low complexity AI Computational CMOS Image Sensors for Spatial Features Extraction

Abstract

The promise of enhanced system performance through the integration of sensing and computing functionalities has necessitated the development of a class of CMOS Image Sensors (CIS) known as Computational CMOS Image Sensors (C^2 IS). These image sensors are capable of multiple on-chip execution of image processing algorithms that include Convolutional Neural Network (CNN). Through on-chip implementation of these algorithms, features can be extracted by the edge device to minimize data movement and processing. This dissertation begins with how to improve the quality of extracted spatial features, then a study of power consumption as it pertains edge devices, and finally, three CMOS Image Sensors (CISs) are proposed for extracting spatial features using low complexity architectures in the analog-domain. In the realm of algorithms, the issue of speckle noise in DoFP polarization images is particularly prevalent, especially in medical imaging applications. In response to this challenge, an adaptive technique for filtering out speckle noise is proposed. Experimental outcomes, comprising both visual representations and numerical analyses, demonstrate the efficacy of the proposed adaptive denoising algorithm in filtering speckle noise from DoFP polarization images, including those present in medical images. The subsequent work focuses on a compression-based architecture for Wireless Vision Sensor Networks (WVSNs), wherein the captured image is divided into subimages, progressively compressed, and thereafter transmitted to a central server equipped with Convolutional Neural Networks (CNNs). The model aims to investigate the balance between power consumption and transmission bandwidth. The findings reveal the necessity of striking a compromise between maximizing edge computation and minimizing power usage. A significant insight from the study is the dominance of transmission power over processing power consumption. Consequently, a need to extract spatial features at the edge is established and three CMOS Image Sensors are subsequently designed for that purpose. The first CIS is designed for real-time edge detection. It introduces an analog domain convolution approach utilizing an extremely low-complexity two-transistor (2T) pixel architecture. This convolution scheme is designed to implement the Laplacian filter for omnidirectional edge detection (across eight feature directions) in the analog domain. It is notable for featuring the simplest pixel architecture, resulting in enhanced Fill Factor and scalability. The second CIS is a multi-mode C^2 IS capable of performing various operations, including execution of the convolution operation of the first layer of a Convolutional Neural Network (CNN). While retaining the low-complexity 2T pixel architecture, this multi-mode C^2 IS enables the execution of the first CNN convolution layer to extract spatial features with up to 8 programmable convolution filters. In addition to reporting the lowest pixel complexity, post-layout simulation results demonstrate that the proposed C^2 IS can conduct convolution with 8 filters while consuming merely 95μW at 72 frames per second (fps), with an efficiency rating of 1.67TOPS/W, a performance comparable to state-of-the-art implementations. Last but not least, the third CIS represents a programmable intelligent C^2 IS capable of executing multi-layer Lightweight Convolutional Neural Networks (LWCNNs) directly on-chip. This intelligent C^2 IS conducts convolution within a 3 × 3 window, incorporates rectified linear unit (ReLU) activation, performs max pooling, and ultimately provides a binary decision through a digital fully-connected layer. With an operational speed of 125 frames per second (fps), this intelligent C^2 IS boasts a power figure-of-merit of 45 pJ/pix·fps. Like its predecessors, it features the simplest pixel architecture.

Date of Award	2024
Original language	American English
Awarding Institution	HBKU College of Science and Engineering