Sage Journals: Discover world-class research

Abstract

This work discusses an innovative approach to imaging which can improve the robustness of existing active-range measurement methods and potentially enhance their use in a variety of outdoor applications. By merging a proven modulation technique from the domain of spread-spectrum communications with the bleeding-edge CMOS sensor technology, the prototype of the modulated range sensor is designed and evaluated. A suitable set of application-specific spreading codes is proposed, evaluated and tested on the prototype. Experimental results show that the introduced modulation technique significantly reduces the impacts of environmental factors such as sunlight and external light sources, as well as mutual interference of identical devices. The proposed approach can be considered as a promising basis for a new generation of robust and cost-efficient range-sensing solutions for automotive applications, autonomous vehicles or robots.

Keywords

range sensing modulated imaging CDMA

1. Introduction

Over recent years a lot of effort has been devoted to research and development of range sensors. These are widely used in numerous applications in medicine, industry, defence, automotives, robotics, and home entertainment. The majority are based on active optical-range measurement techniques [1, 2], which involve the projecting of structured light patterns onto a scene with further reconstruction of the range map by measuring the temporal or spatial distribution of the reflected light. These methods are efficient in controlled disturbance-free environments. The lack of cost-efficient and robust solutions suitable for work in real-world environments motivated our prior work [3, 4], where we introduced the concept of modulated acquisition of spatial distortion maps-snapshots of multi-line patterns projected onto the object. These patterns can be converted to 3D mesh according to the laser triangulation [5] principle as depicted in Figure 1: the distance to the point O(x,y,z) belonging to the pattern's projection at the object's surface is calculated from fixed parameters of the detector (focal length f, base line b between the optical centre of the lens and the laser source, angle α between the base line and the plane of the projected light) and the position of point P(v,u) belonging to the pattern's reflection on the sensor plane:

$[x, y, z] = \frac{b}{f ctg (α) - u} [u, v, f]$ (1)

Figure 1.

Laser triangulation principle and modulated imaging system scheme

Thus, distance detection in this method relies on determining the coordinates of the illuminated pixels and the index of line (i.e., angle α) they belong to.

The classical approach is to try to detect the illuminated pixels in plain snapshots taken from the sensor. This approach is sensitive to external light sources which can overpower the laser and cause erroneous results. Our improvement (pixel intensity modulation controlled by pseudo-random binary sequences) makes pattern detection more robust. Pattern parsing, i.e., robust segmentation and indexing of the detected lines, is a subject for further study. Promising approaches to 3D reconstruction based on multi-line patterns are considered in [6, 7]. In this paper we focus only on the performance of modulated imaging systems, which should produce sharp and accurate images of the projected patterns used as input data for further 3D transformation that is beyond the scope of this paper.

In prior work we showed the efficiency of using cyclically orthogonal Walsh-Hadamard codes (COWHC) [8, 9] while addressing the impacts of the external light sources as well as some limited impacts of mutual interference. The main disadvantage of this code set is the limited number of coexisting units in a system. We showed that for a Walsh matrix of size N×N where N=2^K there were only K codes suitable for our system. This limitation is unacceptable for systems with a flexible number of units, as is desirable in automotive applications or robotics.

In this work we propose suitable system-specific spreading codes in order to extend the system capacity. Furthermore, we demonstrate the application of the selected codes in the real environment.

The paper is organized as follows: in Section 2 the model of modulated imaging system is introduced. In Section 3 the system-specific requirements for modulation codes are identified, while the proposed codes' design and performance evaluation methodology are presented in Section 4. In Section 5 the experimental set-up and conditions are given. In Section 6 the experimental results are demonstrated. Finally, the concluding remarks and further work are given in Section 7.

2. The concept of modulated active imaging

Essentially, active optical range sensors are advanced devices which emit electromagnetic energy in the visible or IR spectrum and utilize the reflected energy for 3D reconstruction of the scene surface. Range data (or 3D mesh) is recovered from the distorted reflection of the patterns according to the physical principle of the sensor (triangulation, time-of-flight [10], interferometry [11], etc.).

Let us define active imaging system as a subsystem of an active range sensor (or a device with a similar purpose) which performs the dedicated task of capturing images of the patterns projected on the scene.

Such a system can be built upon a CCD or CMOS sensor supplemented with an electronically triggered laser as depicted in Figure 1. Both laser and sensor may operate in either the visible or the IR spectrum. A diffractive optical element is installed at the laser's emitter in order to form a desired pattern. The laser is triggered synchronously with the electronic shutter of the sensor when the snapshot is being taken.

The robustness of such an imaging system (i.e., its ability to produce sharp images of the projected pattern) is directly related to the laser's power. Unfortunately, increasing the laser's power is problematic because of high energy consumption and eye safety issues. In addition, the high laser power does not solve the problem of mutual interference from nearby systems. Better results can be achieved with an identical or even a weaker laser by applying a sequential frame acquisition approach: the frame is assembled from the per-pixel sum of N sub-frames taken consequently with the laser turned on or off according to the binary sequence c∊{0;1}as follows:

$I = \sum_{i = 0}^{N - 1} (b + c_{i} l) v_{i}$ (2)

where I is the resulting intensity of the distinct pixel of the image, b is mean background intensity, ι is laser's contribution, and v is modulation sequence c transformed to bipolar form {−1; 1} as follows:

$v_{i} = - {(- 1)}^{c_{i}}$ (3)

Thus, pixels of inactive sub-frames (c_i = 0, v_i = −1) are taken with negative sign while pixels of active sub-frames (c_i = 1, v_i = 1) are taken with positive sign. For ease of further analysis, consider b and ι constant through the frame integration cycle. With the balanced sequence c (even length N, number of ones equal to number of zeros), we can expect that only a pattern itself will be present at the resulting image, while the background and the objects will be suppressed completely as follows:

$I = {\begin{matrix} l \frac{N}{2}, & s i g n a l p i x e l s \\ 0, & b a c k g r o u n d \end{matrix}$ (4)

The greater the parameter N, the better the contrast between the illuminated (signal) and the background (noise) pixels.

A similar effect of raising the signal-to-noise ratio (SNR) is known in direct sequence spread spectrum communications as processing gain. Recent studies [12–14] show that the performance of such systems, i.e., their robustness to external factors and mutual interference, is mainly determined by the correlation properties of modulation codes.

3. Active imaging system-specific requirements for modulation codes

Our imaging system can be considered as an asynchronous CDMA system. When multiple active range detectors scan the same scene simultaneously, each one should be able to filter out the pattern it projects from those projected by the other devices. This case resembles telecommunications systems where multiple transmitter-receiver pairs communicate simultaneously over the shared communication media (wire, radio frequency band or optical frequencies). In order to separate the desired signal from the interference, each transmitter “spreads” the data signal over the wider frequency band by time multiplexing with the unique binary sequence of higher rate. The receiver recovers the original signal by multiplying the observed noise-like signal with the same sequence. If the system is synchronous, i.e., all the transmitter-receiver pairs are coordinated, the best choice is orthogonal codes, since they provide zero in-phase cross-correlation and thus eliminate the multiple access interference (MAI) completely. But if the units of the system are autonomous (as in our case), the interfering signals may intrude at arbitrary random times and will not always be suppressed. Therefore, another approach is required.

Consider K identical-range detectors scanning the same scene. All of them are equally distanced from the surface, have equal laser power and the same predefined modulation sequence c of period N. For simplicity assume all the pixels' intensities induced by any single laser are equal to the same level ι. Before proceeding with the analysis, let us introduce a function R_BU, which represents the periodic correlation of bipolar sequence v with its unipolar form c:

$R_{B U} (τ) = \sum_{i = 0}^{N - 1} c_{i - τ} v_{i}$ (5)

The interference as the resulting pixel intensity at the hypothetical intersection of all interfering projections (as seen by the distinct detector) is evaluated by introducing the contribution of K−1 laser intensities ι triggered by c cyclically shifted by τ to eq. (1):

$I_{M A I} = \sum_{i = 0}^{N - 1} (b + \sum_{k = 1}^{K - 1} c_{i - τ_{k}} l) v_{i} = l \sum_{k = 1}^{K - 1} R_{B U} (τ_{k})$ (6)

It is self-evident that in order to minimize i_MAI we should find sequences with optimal bipolar-unipolar correlation, i.e., with constantly low or zero magnitudes of R_BU at any non-zero shift. It is also important that these sequences must be balanced to suppress the background.

Ideally, modulation sequence should exhibit a specific type of perfect auto-correlation property, i.e., have zero out-of-phase bipolar-unipolar correlation at any non-zero shift. In this case, according to Equation 6 the interference is eliminated completely. The performance of such a system could be evaluated in terms of the probability of in-phase operation with one or more interfering devices, or in other words, frames collision probability (P_FC):

$P_{F C} = (K - 1) \frac{N_{p}}{N}$ (7)

where N_p is the number of peaks of correlation function R_BU (Equation 5), which equals 1 for perfect sequences.

Frame collisions can be detected in several ways during image processing. For example, a drastic rise of “white level” in a histogram or unexpected number of lines in multi-line pattern could be detected. Efficient error-recovery strategy in this case involves dropping the erroneous frame and introducing a random (several sub-frames long) delay before restarting the frame integration cycle.

If the sequence is not perfect but provides at least a constantly low level of out-of-phase correlation, it can be considered optimal. In addition to frame collision probability these sequences could also be characterized by peak-to-average ratio (PAR):

$P A R = \frac{R_{B U} (0)}{R_{L C Z}},$ (8)

where R_LCZ is the average of absolute values of R_BU for the low correlation zone. According to the definitions of signal (2) and interference (5), and provided that all interfering devices contribute equal laser intensity ι, we can estimate signal-to-interference ratio (SIR) as follows:

$S I R = \frac{N}{2 (K - 1) R_{L C Z}} = \frac{P A R}{K - 1}$ (9)

Error probability P_E for the discussed system modulated by the optimal sequence can be extended when spatially superposed interference overwhelms the signal:

$P_{E} = {\begin{matrix} P_{F C}, & f r a m e c o l l i s i o n \\ 1, & S I R \leq 1 \end{matrix}$ (10)

The recommended approach for minimizing this error involves introducing a proper threshold for pixel values below the accepted noise level before detecting the frame collisions. A recovery strategy is to extend N, thus improving SIR, which should be combined with frame collision recovery.

4. Design and analysis of optimal modulation code for the imaging system

Hunting for perfect sequences, i.e., sequences with zero out-of-phase periodic auto-correlation function (ACF), has been a subject for many studies related to CDMA applications. Some general approaches to construction of complex-valued sequences with the desired properties have been proposed recently [15, 16]. Unfortunately, there are no {−1;+1} sequences with perfect ACF of length 4 > N ≥ 12100 [17].

We examined some known modulation codes with almost-perfect ACF, including Barker codes [18] and Wolfmann codes [19], and concluded that none of them satisfy our specific criteria for both optimal periodic bipolar-unipolar correlation and balancing.

However, m-sequences were found to be promising candidates because of their perfect R_BU function (see Figure 2).

Figure 2.

Bipolar-unipolar correlation (R_BU) of 31-bit-long m-sequence

The only challenge is balancing, since the binary m-sequences are of odd-length N=2^p−1, where p is a prime number.

Our approach to code balancing uses Manchester encoding: each “1” symbol of the original sequence is replaced with the combination of “−1;+1” while “−1” (or “0”) is replaced with “+1;−1”. By applying this transform to an m-sequence of length m one can obtain an optimal balanced sequence of length N=2m with parameters R_LCZ=1, PAR=N/2. Periodic auto- and bipolar-unipolar correlation function of a 62-bit-long Manchester-encoded m-sequence (MEMS) derived from an original 31-bit m-sequence is plotted in Figure 3.

Figure 3.

Bipolar-unipolar correlation (R_BU) of obtained 62-bit-long MEMS sequence

Obtained MEMS sequences can be compared to COWHC codes of similar widths evaluated in prior work. According to conducted numerical simulations the newly proposed design based on MEMS sequences can accommodate more units than the previous design until saturation limit is reached – in other words, while the acceptable level of error probability P_E is provided. This limit designates the system capacity characteristic by showing how many sensors may operate simultaneously without “frame collisions” with one or more nearby sensors. For the MEMS-modulated system we define this limit as P_FC ≥ 0.5, which means that the system can tolerate up to 50% frame drops in the worst case while trying to apply error recovery.

Low SIR as another error case (according to Equation 10) in normal (out-of-phase) operation of MEMS codes may be caused only by spatial superposition of several projections. We assume that this condition is hardly possible in real applications (for example in robotics) because the units' placement in real terrain cannot lead to the ideal superposition of multiple projections.

We performed computer simulations to compare P_E values for a 16-bit-long COWHC sequence vs. a 14-bit-long MEMS, a 32-bit-long COWHC sequence vs. a 30-bit-long MEMS, and a 64-bit-long COWHC sequence vs. a 62-bit-long MEMS. We found that saturation limit (P_FC ≥ 0.5) for MEMS codes was reached at number of users K=N/2, while the error condition for COWHC was reached at K=log₂N.

The simulation results are depicted in Figure 4, showing that the capacity of the system using MEMS codes outperforms the system with COWHC codes. The difference in system capacities is increased by increasing the number of bits in the sequences.

Figure 4.

Error probability (P_E) vs. number of users for COWHC and MEMS codes of similar widths

5. The experiments

In the experiments we demonstrate the background light and the mutual interference suppression effects, described in the previous chapters.

The experimental set-up demonstrating the discussed principle of modulated imaging is depicted in Figure 5. It was built around a commercially available industrial camera (Velociraptor, Optomotive Mechatronics Ltd, Ljubljana, Slovenia) [20] based on a high-speed CMOS sensor (CMOSIS CMV2000). The camera makes a series of snapshots at a high frame rate (up to 600 fps) with a short exposure time. A specifically programmed FPGA processing core performs in-camera frame integration as discussed in Section 2. Each pixel of the frame is calculated as a sum of the corresponding pixels of N sub-frames taken with positive or negative sign according to the highest bit of the modulation sequence cyclically shifted in the modulation register. The laser is switched ON or OFF according to this bit. With optimal modulation sequences the resulting pixel intensities are expected to be as evaluated by Equations 4 and 6, i.e., near-zero values (black levels) for the suppressed background, ambient light and interference, while the illuminated pixels are expected to be high-level values (grey and white levels).

Figure 5.

Experimental setup

The scene is illuminated using a modulable 650 nm LED laser combined with a DOE (diffractive optical element) projecting a pattern of parallel horizontal lines. The resulting frame rate for the 30-bit-long modulation code is ca. 22–24 fps.

During the experiments we took a series of snapshots from the live view of the camera in different illumination conditions and with the presence of a nearby detector.

The background light suppression experiment was conducted indoors in normal daylight conditions first. Next, the experiment was conducted outdoors in sunlight.

A mutual interference suppression experiment was conducted indoors with the presence of a second detector. In the first part of this experiment the detectors were modulated by a non-optimal pair of orthogonal codes. In the second part of the experiment both detectors were modulated by the same MEMS code proposed in this work.

6. Results

In the first series of the results we demonstrate the background light suppression effect of the proposed active imaging system. Snapshots from the indoor live footage are shown in Figure 6, while the snapshots taken outdoors are shown in Figure 7.

Figure 6.

Backgroung light suppression effect indoors. Plain-view snapshot (up) and demodulated-view snapshot (down).

As expected, a remarkable effect of real-time background suppression is achieved – only the pattern itself is present in the live footage, while the objects and the background are suppressed completely.

In the latter case it is clearly seen that even if laser illumination seems completely overpowered by the sunlight and is not clearly visible to the naked eye, the introduced approach allows us to amplify the tiny differences induced by the laser such that the resulting image captured outdoors (Figure 7) is of satisfactory quality for further pattern recognition.

Figure 7.

Background light suppression effect outdoors. Plain-view snapshot (up) and demodulated-view snapshot (down).

The second series of results demonstrates mutual interference of the two detectors currently available to us. Figure 8 demonstrates the influence of a nearby detector when the detectors are modulated by a non-optimal pair of orthogonal codes. In the demodulated-view snapshot the interference is clearly observed. Visually, a similar effect is produced by any other non-optimal pair of modulation codes, or when optimal MEMS codes operate in-phase (synchronously activated).

Figure 8.

Interference of nearby detector with non-optimal modulation code. Plain-view snapshot (up) and demodulated-view snapshot (down).

Finally, we tested the proposed MEMS modulation codes and achieved satisfactory results with out-of-phase operation. There is no noticeable interference present in the live footage (Figure 9) when the MEMS code operates normally (out-of-phase) with the applied threshold.

Figure 9.

Interference of nearby detector with optimal modulation code. Plain-view snapshot (up) and demodulated-view snapshot (down).

7. Conclusion

This work has discussed the principle of modulated active imaging and its non-orthogonal design. The proposed approach can be considered optimal for development of fail-tolerant self-adapting active sensing systems with non-fixed number of units when it is not technically possible to coordinate the system centrally or provide each unit with a unique predefined modulation code. With the proposed approach, a greater number of users (i.e., greater system capacity) can be achieved than with the prior (orthogonal) design. A possible disadvantage of the proposed solution is the requirement for thresholding the low-intensity pixels, which may be induced by partial (in-chip) coincidence with the main peak or secondary peaks of correlation with the interfering signal, and thus may lead to loss of density of distant objects in the resulting range map. However, it can still solve the application-critical task of robust detection of the nearest obstacles in the presence of interfering devices. Improvement of the proposed system may be achieved by increasing the length of the modulation sequence or searching for sequences with better bipolar-unipolar correlation properties.

Footnotes

8.

This research was supported by the European Union,European Social Fund,within the scope of the framework of the Operational Programme for Development of Human Resources in the Period 2007–2013 contract No. 3211-11-000492.

References

Sansoni

Trebeschi

Docchio

State-of-the-art and applications of 3D imaging sensors in industry, cultural heritage, medicine, and criminal investigation. Sensors. 2009; 9:568–601. DOI: 10.3390/s90100568.

Martínez-Gómez

Fernández-Caballero

García-Varea

Rodríguez

Romero-González

A Taxonomy of Vision Systems for Ground Mobile Robots. Int. J. Adv. Robot. Syst. 2014; 11:111. DOI: 10.5772/58900.

Volkov

Gros

J.Ž.

Žganec

Javornik

Švigelj

Modulated Acquisition of Spatial Distortion Maps. Sensors. 2013; 13:11069–11084. DOI: 10.3390/s130811069.

Zganec

J.G.

Active 3D Triangulation-Based Method and Device. 2009; US Patent 7,483,151B2

Tawfiq

Abouei

Plataniotis

K.N.

Cyclic Orthogonal Codes in CDMA-Based Asynchronous Wireless Body Area Networks. In: IEEE International Conference on Acoustics, Speech and Signal Processing; 25–30 March 2012; Kyoto. New York: IEEE; 2012. p. 1593–1596

Molder

Martens

Saar

Land

Adaptively undersampled image processing for fast multiline laser detection. In: Intelligent Signal Processing. Proceedings of IEEE 8th International Symposium; 16–18 Sept. 2013; Funchal. New York: IEEE; 2013. p. 60–64; DOI: 10.1109/WISP.2013.6657483

Gao

Zhou

Peng

Wang

, and Tan

3D Wide FOV Scanning Measurement System Based on Multiline Structured-Light Sensors. Advances in Mechanical Engineering. 2015; 6:758679–758679; DOI: 10.1155/2014/758679.

Abouei

A set of cyclic orthogonal codes acquired from Walsh-Hadamard matrix. In: Proceedings of 34th International Mathematics Conference; Shahrood; 2003.

Kawahito

Halin

I.A.

Ushinaga

Sawada

Homma

Maeda

A CMOS time-of-flight range image sensor with gates-on-field-oxide structure. IEEE Sens. J. 2007; 12:1578–1586

10.

Franca

J.G.D.M.

Gazziro

M.A.

Ide

A.N.

Saito

J.H.

(2005) A 3D Scanning System based on Laser Triangulation and Variable Field of View. In: Proceedings of IEEE International Conference on Image Processing; 11–14 September 2005; Genoa; New York: IEEE; 2005. vol. 1, p. 425–428

11.

Thiebaut

Giovannelli

J.F.

Image reconstruction in optical interferometry. IEEE Signal Process Mag. 2010; 1:97–10

12.

Khan

A.R.

Performance evaluation of DS-CDMA system using MATLAB. Int. J. Adv. Eng. Technol. 2012; 6:269–281

13.

Khatter

Goyal

Design and analyze the various m-sequences codes in MATLAB. Int. Emerg. Technol. Adv. Eng. 2012; 11:125–129

14.

Ravindrababu

Krishna Rao

E.V.

Performance comparison of spreading codes in linear multi user detectors for DS-CDMA system. WSEAS Trans. Commun. 2013; 2:52–62

15.

Blake

S.T.

Tirkel

A.Z.

A Construction for Perfect Periodic Autocorrelation Sequences. In: Schmidt

Winterhof

, editors. Sequences and Their Applications – SETA 2014. Proceedings of 8th International Conference; 24–28 November 2014; Melbourne; New York: Springer; 2014. p. 104–108; DOI: 10.1007/978-3-319-12325-7.

16.

Chen

Zhao

Huang

Zhu

Almost perfect sequences applied for ionospheric oblique backscattering detection. Radio Sci. 2009; vol. 44; DOI: 10.1029/2009RS004148.

17.

Baumert

L.D.

Cyclic difference sets. In: Lecture Notes in Mathematics; New York: Springer-Verlag; 1971. vol. 182

18.

Barker

R. H.

Group Synchronizing of Binary Digital Sequences. In: Jackson

, editor. Communication Theory. London: Butterworth; 1953. p. 273–287

19.

Wolfmann

Almost perfect autocorrelation sequences. In: IEEE Transactions on Information Theory; New York: IEEE; 1992; vol. 38, no. 4, p. 1412–1418; DOI: 10.1109/18.144729.

20.

Optomotive Velociraptor Camera datasheet [Internet]. 2014. Available from: http://www.opto?motive.com/products/velociraptor Accessed on 19 Mar 2014.