Sage Journals: Discover world-class research

Abstract

Tremendous effort has been devoted toward developing automated post-earthquake inspection techniques, including automated image collection and damage identification. However, few studies have attempted to establish the complex relationship between visible damage and structural conditions. Moreover, the lack of training data further hinders the potential use of deep learning algorithms. This paper proposes a framework, termed Bidirectional Graphics-based Digital Twin (Bi-GBDT), that allows for assessment of the structural condition based on post-earthquake photographic surveys. The procedure contains two parts: (i) A GBDT of a target structure, containing a computer graphics model and a finite element (FE) model, is developed (ii) Synthetic data subsequently is generated from the GBDT to train neural networks to predict the damage measures and structural conditions. To demonstrate the proposed approach, a seismically-designed reinforced concrete shear wall is considered. First, the GBDT of the shear wall is constructed and calibrated so that the associated damage patterns simulated by the corresponding FE model match well with the experimental results. Next, synthetic images of the damage patterns are created from the validated GBDT, along with the corresponding structural damage measures, and used to train Residual Neural Network and Conditional Generative Adversarial Networks to determine the maximum drift, the stress/strain fields and the structural condition. The neural networks trained on synthetic data are shown to perform well for the experimental data, confirming the proposed approach. Subsequently, the neural networks are tested on the synthetic data for a wide variety of loading conditions to demonstrate the robustness of the approach. In addition, the realistic images rendered from the GBDT are utilized as input to predict the structural condition, showcasing the comprehensive Bi-GBDT framework. These results demonstrate the efficacy of the proposed approach to generate accurate digital twins and point toward its future application for development of automated post-earthquake assessment strategies.

Keywords

Synthetic data deep learning computer vision digital twin seismic damage estimation finite element analysis

Introduction

Rapid assessment of the safety of structures after an earthquake is critical for making decisions regarding needed repairs and resumed occupancy/operations. Post-earthquake inspection must evaluate the current safety of affected structures, as well as their safety against future hazards (e.g., aftershocks). Typically, such building assessments are conducted on-site by experts, with the result being that the buildings are classified into three categories: (1) inspected (green), (2) restricted use (yellow), and (3) unsafe (red).^1,2 Such inspections are laborious, time consuming, and subjective.³ Moreover, inspectors are often required to conduct inspections in high-risk situations due to the state of the building being unknown.

To address the shortcomings of traditional post-earthquake inspection approaches, researchers have proposed to automate the process.^4,5 An autonomous inspection workflow can be summarized in three steps: (1) data acquisition, (2) damage identification, and (3) estimation of structural status regarding observed damage. Post-earthquake data is often in the form of images of the damaged structure. Automated image acquisition can leverage advanced unmanned aerial vehicles (UAVs). UAVs allow for efficient access to hard-to-reach regions and facilitate comprehensive data collection. The flight path for a data acquisition task is usually designed in advance, considering environmental conditions and targets of interest.^6–8 For rapid post-earthquake assessment, strategies have been proposed to optimize the acquisition of damage-relevant images from bridges using UAVs, minimizing both time and cost.^9,10 These methods allow for a wealth of images of the structure to be collected autonomously.

Having acquired the images of the structure, researchers have proposed autonomous damage detection methods using computer vision (CV) and deep learning (DL) techniques. Convolutional neural network was employed to detect and localize cracks on a damaged surface.^11–13 Moreover, damage detection methods are extended to process multiple categories of damage types, including surface cracking, spalling, exposed rebar, and buckled rebar.^14–16 Spencer et al.¹⁷ provide a comprehensive review of the extensive literature in this area.

Hoskere et al. developed workflows that combined post-earthquake data acquisition and damage identification.¹⁸ For example, after the 2017 Mexico City earthquake, they proposed the use of fully convolutional neural networks to identify the damage in a dataset acquired from post-earthquake UAV inspections following pre-planned flight paths. Subsequently, the identified damage is superimposed onto a 3D point-cloud model of the entire building, termed a condition-aware model. However, earthquakes are rare events and having the opportunity to conduct a UAV survey of a damaged structure after an earthquake can be challenging.

To address this problem, synthetic environments have been proposed to provide a platform for development of post-earthquake inspection strategies. The concept of physics-based graphics models (PBGM) was proposed which can generate 3D damaged graphic representations of a target structure and associated environment.^19,20 The PBGM is comprised of graphic models based on finite element (FE) analysis, in which the entire data acquisition and damage identification process can be simulated in the computer. Narazaki et al.^21,22 developed a synthetic environment for post-earthquake inspection of high-speed railway viaducts for component recognition, damage identification, generation of a synthetic-real mixed dataset, and autonomous UAV navigation planning. The concept of the PBGM was extended to a Graphics-based Digital Twin (GBDT), which focuses on a specific as-built structure and can be used to predict damage under specified earthquake loading using FE analysis.²³ The PBGM concept was combined with a performance-based earthquake engineering approach to determine damage states from FE analysis.²⁴ The synthetic environments provided by the PBGM and GBDT represent significant strides toward developing the workflow process for automated post-earthquake inspections.

However, the interpretation of the identified damage to yield the overall condition of structures, that is, for the third step of the automated inspection framework, can be subjective and error-prone.^25,26 The complex relationship between earthquake characteristics, structural design, and earthquake-induced damage was emphasized.²⁷ The unpredictability of structural responses was investigated, particularly for critical infrastructures, during seismic events.²⁸ Moreover, some studies focus on specific types of damage. For example, structural cracks tend to close when subjected to cyclic loading, leading to the reported crack widths that correlate poorly with damage.²⁹ Establishing the relationship between identified damage and the overall condition of structures is still needed.

Because inferring the overall status of a structure from identified damage is an “inverse problem” with numerous influence factors, many difficulties arise when trying to solve this problem using traditional physics-driven methods such as FE analysis.³⁰ The inverse problem hereinafter refers to a kind of problem which aims at predicting the “reasons” based on the “results.” Traditional methods, such as the FE method, are not good at solving inverse problems. In recent years, advances in CV and DL technologies have provided novel solutions for addressing such inverse problems.³¹ However, few existing studies have focused on this field, partially due to immature models and very limited data, especially from those models comprehensively validated against experiments.

Therefore, the goal of this research is to develop an experimentally validated framework, termed Bidirectional Graphics-based Digital Twin (Bi-GBDT), to establish the connection between structural conditions and captured damage patterns from the post-earthquake photographic survey. In the proposed approach, a GBDT of the target structure is first established and validated by comparing the experimentally obtained damage maps derived from photographs of the damaged structure with those derived from FE model results. Subsequently, the validated GBDT is used to obtain both the simulated damage map and the corresponding damage measures (e.g., maximum drift, stress/strain fields, etc.) for various loading conditions. This synthetic data is then used to train neural networks to predict damage measures and structural conditions from the input damage map. To demonstrate the proposed approach, a concrete shear wall, tested at the University of Illinois Urbana-Champaign, is considered and compared with both the results of FE model and the prediction using neural networks.³² Moreover, realistic images rendered from GBDT are employed as input to predict the structural condition, showcasing the comprehensive Bi-GBDT framework. This research brings the generation of accurate digital twins one step closer to being able to provide automated post-earthquake assessment strategies based on photographic surveys.

Methodology framework

Figure 1 provides an overview of the framework proposed in this research, termed Bi-GBDT. Fundamental to the proposed approach is the GBDT, a synthetic model that fuses the photorealistic depiction of as-built structures with the corresponding FE model.²³ First, the case of predicting the damage patterns of the structure based on the prescribed loading, termed herein as forward prediction, is considered. To this end, the GBDT of the target structure is created based on detailed design information. Subsequently, the FE analysis is conducted subjected to the prescribed earthquake, and damage measures (e.g., deformation, stress, strain, and damage indices) are obtained. This information is then used to create the damage map, from which images of the damage are obtained and projected onto the 3D graphics model of the structure. Noting that the accurate location of damage is sourced to the associated FE, thus the projection can be easily finished based on the geometric information of the FE model. The forward prediction process was demonstrated for a seismically excited five-story apartment building and explored the implementation of UAV trajectory planning strategy in the synthetic environment.

Figure 1.

Research framework of Bi-GBDT.

The inverse process, termed herein as backward prediction, seeks to determine the damaged state of the structure from images of the damaged structure. This process is divided into two parts: (i) identifying damage maps from images of damaged structures and (ii) predicting the damage state based on the damage maps. Many researchers have developed approaches to address the first component of the backward prediction process, often termed damage identification.^13,33,34 However, little work has been reported on how to determine the state of the structure from the identified damage, which is the central focus of this research.

One of the main roadblocks to assessing the state of the structure from the identified damage is that a mechanics-based approach that provides the connection between the damage maps and the damage measures (i.e., the inverse process) is not apparent. DL approaches show great potential in solving high-dimensional inverse problems, thus offering one possibility for representing such complex processes. However, adequate modeling methods are immature, and available training data is limited.

Therefore, this research proposes the use of forward prediction to create large-scale datasets of realistic damaged structures that can be used to train a neural network for backward prediction. The next section presents a seismically designed concrete shear wall tested at the University of Illinois Urbana-Champaign that will be used to illustrate the proposed research.³² The GBDT for the shear wall, including the FE model, is then developed and validated. Section “Generation of GBDT using forward prediction” introduces the workflow for forward prediction and its associated application, which describes the creation of damaged GBDT.

Subsequently, the backward prediction process to determine damage measures and structural conditions will be introduced in section “Backward prediction based on DL networks,” which is trained on the synthetic data generated by the GBDT.

Seismically designed reinforced concrete shear wall

To clearly illustrate the Bi-GBDT framework, a seismically designed concrete shear wall, tested at the University of Illinois Urbana-Champaign, is selected as the prototype to exemplify the workflow of the framework.³² First, the detailed design of the experiment, including the loading condition and shear wall configuration, are described. Subsequently, an FE model of the wall is constructed using Abaqus, and the corresponding responses are compared with those from the previous experiments.³⁵ The main criteria for comparing the FE model and experimental results are (i) base moment, (ii) drift, and (iii) damage patterns. After validating the effectiveness of the established FE model, accurate synthetic data can be generated through FE simulation to train the DL networks employed in the backward prediction.

Experiment setup

The prototype structure is a modern 10-story shear-wall building that is typical for regions with high seismicity. The specimen tested in laboratory represents the shear wall corresponding to the first three stories of such a 10-story building. Loading is applied to induce the wall to behave as if it is tested as part of the full 10-story high shear wall. As shown in Figure 2(a), a transfer beam is placed on top of the wall cap and connected to the two load and boundary condition boxes. The transfer beam is assumed to be rigid, resulting in the top surface of the cap being undeformed (i.e., only exhibiting rigid body motion). Thus, the associated axial load $N$ , lateral load $V_{app}$ , and overturning moment $M_{app}$ (see Figure 2(b)) corresponding to the top seven stories are applied to the top of the wall specimen via this transfer beam attached to the wall cap. The lateral displacement was specified at the control point, which is located on the top center of the specimen, and associated shear force was measured. The overturning moment was commanded such that its ratio to the shear force was constant at 5 m. The displacement control protocol applied in the experiment is displayed in Figure 3.

Figure 2.

Loads on the specimen: (a) experiment setup³⁶ and (b) simplified loads.

Figure 3.

Horizontal displacement control protocol: (a) horizontal displacement and (b) applied loads.

FE model configuration

A FE model is developed in Abaqus corresponding to the shear wall specimen. Figure 4 presents the dimension and reinforcement design for the wall specimen.³⁶

Figure 4.

Reinforced concrete wall design³⁶: (a) elevation and (b) cross-section.

The shear wall is modeled in three sections (i.e., the wall cap, the base foundation, and the wall) using 3D solid element (C3D8R). All sections are connected together by tie constraints. The reinforcement is modeled with 3D truss elements (T3D2) combined with the 3D solid concrete elements (C3D8R) using an embedded region constraint. A mesh size of 25 mm is chosen to simulate clear and detailed damage. The mesh of the FE model is shown in Figure 5(a). The displacements of nodes on the bottom surface are all fixed, and displacement control protocol shown in Figure 3 is applied on the wall cap. In addition to horizontal displacement, vertical displacement and rotation are applied to the top surface, enabling the accurate reproduction of detailed in-plane deformation.

Figure 5.

FE model of reinforced concrete wall specimen.

Constitutive model

This section describes the Abaqus constitutive models employed for the concrete material and steel reinforcement.

Concrete

The concrete damage plasticity model (CDPM) available in Abaqus is employed to simulate concrete behavior in this study. The stress–strain relationship must be defined in the CDPM for the uniaxial behavior of the concrete for both tension and compression. The uniaxial compressive behavior of concrete proposed by Hognestad³⁷ is used herein and given by

$σ_{c} = σ_{cu} [2 (\frac{ε_{c}}{ε_{cu}}) - {(\frac{ε_{c}}{ε_{cu}})}^{2}]$ (1)

where $σ_{c}$ and $σ_{cu}$ denote nominal compression stress and ultimate compressive stress, respectively, and $ε_{c}$ and $ε_{cu}$ denote nominal and ultimate compressive strain, respectively. The associated compressive uniaxial material properties of concrete used for the wall specimen are obtained from three cylinder tests.³⁶ The data from the compressive cylinder, along with the CDPM stress–strain curve used, are compared in Figure 6. The uniaxial tensile behavior is assumed to have linear elastic and exponential tension softening regions.³⁸

Figure 6.

Stress–strain curves of concrete model and three-cylinder tests.

The tensile strength and strain of concrete in the experiment are measured from modulus of rupture tests. Also, plasticity parameters are required to develop the yielding behavior of the CDPM based on its uniaxial behavior.^39,40 Table 1 presents the parameters employed for the CDPM in the FE model.⁴¹ In addition, the CDPM represents damage intensity under cyclic loading by degradation of the elastic stiffness of the concrete, illustrated in Figure 7.

Table 1.

Plasticity parameters of CDPM.

Parameter	Dilation angle	Eccentricity	$f_{b 0} / f_{c 0}$	K	Viscosity parameter
Value	31	0.1	1.16	0.67	0

CDPM: concrete damage plasticity model.

Figure 7.

Response of uniaxial loading of concrete: (a) compression and (b) tension.

The degradation of elastic stiffness illustrated in Figure 7 is characterized by two damage indices, $d_{c}$ and $d_{t}$ , for compression and tension stages, respectively. These indices are defined as:

${\begin{matrix} d_{c} = 1 - \frac{σ_{c}}{σ_{cu}}; 0 \leq d_{c} \leq 1 \\ d_{t} = 1 - \frac{σ_{t}}{σ_{tu}}; 0 \leq d_{t} \leq 1 \end{matrix}$ (2)

where $σ_{t}$ and $σ_{tu}$ are the nominal and ultimate tension stresses, respectively. The initial elastic stiffness $E_{0}$ starts to reduce as the nominal stress decreases in strain softening region, which influences the uniaxial behavior beyond the ultimate stress in each loading cycle. In the CDPM implementation in Abaqus, the stress–strain relationship is expressed as,

${\begin{matrix} σ_{c} = (1 - d_{c}) E_{0} (ε_{c} - ε_{c}^{pl}) \\ σ_{t} = (1 - d_{t}) E_{0} (ε_{t} - ε_{t}^{pl}) \end{matrix}$ (3)

${\begin{matrix} ε_{c}^{in} = ε_{c} - \frac{σ_{c}}{E_{0}} = ε_{c} - ε_{c}^{el} \\ ε_{t}^{in} = ε_{t} - \frac{σ_{t}}{E_{0}} = ε_{t} - ε_{t}^{el} \end{matrix}$ (4)

${\begin{matrix} ε_{c}^{pl} = ε_{c}^{in} - \frac{d_{c}}{1 - d_{c}} \frac{σ_{c}}{E_{0}} \\ ε_{t}^{pl} = ε_{t}^{in} - \frac{d_{t}}{1 - d_{t}} \frac{σ_{t}}{E_{0}} \end{matrix}$ (5)

where $ε_{c}^{pl}$ and $ε_{t}^{pl}$ refer to the equivalent plastic strain; $ε_{c}^{in}$ and $ε_{t}^{in}$ denote the inelastic strain; $ε_{c}^{el}$ and $ε_{t}^{el}$ are the elastic strain. Based on the CDPM introduced in this section, the inelastic behavior and damage index required by Abaqus for both compression and tension are displayed in Figure 8.

Figure 8.

Input of CDPM in Abaqus: (a) Inelastic stress–strain curve in compression, (b) compression damage index, (c) inelastic stress–strain curve in tension, and (d) tension damage index.

Steel

With linear elastic and linear plastic assumptions, the constitutive model of steel is a piecewise linear relationship, as shown in Figure 9, where the material properties are measured from standard tension tests.³⁶ The compressive and tensile behaviors are assumed to be identical to simplify the model.

Figure 9.

Stress–strain curves of steel: (a) No. 2 bar and (b) No. 4 bar.

Comparison with experimental data

To validate the effectiveness of proposed FE model, the numerical FE results are compared with the experimental results. Both load–displacement response and the associated damage patterns are considered in the comparison. For concrete shear walls in high-rise buildings, drift levels exceeding 0.5% are often associated with obvious and severe damage and can be readily identified.^1,42 Therefore, these circumstances are not considered in this study. Only the more complex and confusing drift ratio ranges (less than 0.5%) are analyzed in this study.

Load–displacement response

The structural response of the proposed FE model is validated using load–displacement responses subject to cyclic loading, depicted as hysteretic curves in Figure 10. As illustrated, the simulation outcomes with the simplified CDPM align closely with the experimental results despite some observable deviation due to the FE model’s use of comprehensive in-plane boundary conditions.³⁶

Figure 10.

Comparison of hysteretic curves between the experiment and FE analysis.

Damage patterns and failure modes

Two different damage indices, which characterize the degradation of elastic stiffness, are produced by the Abaqus analysis, corresponding to tension and compression. If either of the two indices in an element exceeds 0.9, then the element is considered to be damaged. A raw damage map can be obtained by collecting and rearranging the damage indices for each element from Abaqus simulation (Figure 11(a) and (b)). The damage map solely contains the damage indices of elements on the surface, creating a 2D representation of damage patterns to match the format of visual damage obtained from photographic surveys. The resolution of this raw damage map, however, does not adequately represent real-world damage. This is because the minimum mesh size of 25 mm is significantly larger than the actual crack width found in the physical specimen (i.e., less than 3 mm). Using a more refined mesh would lead to an unacceptable computation burden, particularly when establishing the neural network dataset, which demands numerous simulations. Therefore, image post-processing is needed to convert the raw damage map to realistic damage map applied directly to the graphics model.

Figure 11.

Damage maps through post-processing method: (a) Abaqus simulation, (b) raw damage map, (c) skeletonized, and (d) noised.

The proposed image post-processing method involves a four-step process that transitions the low-resolution damage map to a highly detailed realistic damage map: (i) threshold filter, (ii) resize, (iii) skeletonization, and (vi) noise evaluation. Initially, the low-resolution damage map is filtered by a threshold damage index of 0.9, differentiating between genuine damage and undamaged surface. Next, the filtered damage map is resized from $488 \times 584$ to $4880 \times 584$ 0 to reduce the physical size corresponding to a pixel (i.e., minimum crack width). Compared to the size of shear wall (i.e., $3048 \times 3657.6 mm$ ), the resized resolution allows for a detailed representation, capturing cracks with a minimum width of less than 1 mm. The resultant damage map becomes a distinct binary image. Due to the scale of the single damage line compared to the entire image being larger than the realistic crack’s scale relative to the concrete wall, the damage map undergoes skeletonization to eliminate border pixels of the damage, as illustrated in Figure 11(c). The skeletonized damage map provides clear information about the direction and location of damage, maintaining the width and the scale of the damage.

However, the map might display unrealistically straight lines and sharp corners. To remedy this problem, Perlin noise is applied to the skeletonized map to inject realistic detail using the noise texture node available in Blender.^43,44 Four properties govern the noise texture: “Scale,”“Detail,”“Roughness,” and “Distortion.”“Scale” input defines the base noise wave size. “Detail” input outlines the noise’s complexity, ranging from 0–16. Larger “Detail” input renders the noise finer and richer in detail. “Roughness” input modulates the noise’s amplitude smoothness and sharpness, while “distortion” input regulates the number of image-twisting effects. Table 2 outlines the noise properties values utilized in this study. The application of Perlin noise infuses realistic aspects of concrete damage onto the damaged position, maintaining the original locations and directions of damage for authenticity and correctness of the FE analysis-derived damage map. Figure 11(d) exhibits the influence of noise on the skeletonized damage map.

Table 2.

Properties of noise texture node.

Parameter	Scale	Detail	Roughness	Distortion
Value	1.0	16.0	0.75	0

The damage patterns from the FE analysis are compared at six different stages of the experiment, as shown in Figure 12. These stages cover three different drift levels (i.e., 0.139%, 0.208%, and 0.347%) and two peaks (i.e., pushing right and left) of the first cycle at each drift level. The positive and negative drift indicates the deformation toward right and left, respectively. Because the experiment only recorded the bottom two-thirds of the shear wall, the damage map from FE simulation is cropped to the same size.

Figure 12.

Crack pattern comparison: (a) Drift = +0.139% (Step 75), (b) Drift = −0.139% (Step 85), (c) Drift = +0.208% (Step 135), (d) Drift = −0.208% (Step 145), (e) Drift = +0.347% (Step 240) and (f) Drift = −0.347% (Step 260).

As observed from Figure 12, the simulation successfully mimics the experimental damage patterns considering the density, distribution, and direction of cracks for drifts below 0.5%. For instance, at step 75, damage manifests on the left side of the shear wall as a result of the exerted rightward push, with horizontal cracks beginning to emerge along the left boundary element. These cracks extend, with major ones propagating along the shear wall’s left half. By step 85, the right side of the shear wall exhibits similar damage patterns due to the equivalent leftward push. Furthermore, during steps 135 and 145, as the shear wall is subjected to sequential pushes to the right and left at a drift of 0.208%, diagonal cracks start to propagate from the boundary elements toward the base corner in the direction of force. More refined horizontal cracks emerge at the boundary and the middle starts extending from the lower part toward the opposite corner. As the drift extends to 0.347% during steps 240 and 260, the damage escalates, becoming denser. An increase in diagonal cracks is observed at the middle and bottom of the shear wall, accompanied by the emergence of more fine horizontal cracks at the boundary element.

Moreover, pronounced deformation is observed at the splice’s top, which is 609.6 mm above the foundation. This location registers spalling on both sides during the experiment when higher drift levels are applied. The graphical fidelity of the damage patterns, as simulated by the proposed FE model, presents a satisfactory representation of real-world damage observed in a concrete shear wall under cyclic load. Considering the inherent randomness of damage propagation, the consistent alignment of damage patterns in terms of density, distribution, and direction provides reliable physical features and makes it a solid basis for training DL networks for backward prediction in subsequent sections.

Generation of GBDT using forward prediction

In this section, the complete workflow of the forward prediction is illustrated by the creation of damaged GBDT. The damaged GBDT utilizes the information derived from the FE model and incorporates it into the undamaged GBDT through the following five steps: (1) extracting the geometric configuration, (2) damage map generation, (3) model texturing, (4) damage map texturing, and (5) synthetic images rendering, noting that the detail of damage map generation process has been introduced in section “Damage patterns and failure modes.” The forward prediction process for creating a digital twin with the same configuration and realistic damage pattern as the FE model has been automated, opening the possibility of building large-scale datasets and incorporating shear walls with different designs. With the creation of damaged GBDT, a showcase of the integration of forward and backward prediction will be presented section “Backward prediction based on DL networks.” This showcase closes the circle for Bi-GBDT framework by explaining how the synthetic data generated through forward prediction can be utilized to train the backward prediction.

Geometric information

The geometric information regarding the mesh, including element types, geometric relations, and nodal coordinates, is extracted from the FE model. The extraction process uses the Abaqus-Python API to automatically export the large-scale geometric information and FE simulation results. Such information is then channeled into Blender, a 3D creation software, where it is rendered into the GBDT, subsequently enabling the generation of extensive synthetic data.⁴³

The FE model incorporates two types of Abaqus elements: (i) the 3D solid element (C3D8R) for concrete and (ii) the 3D truss element (T3D2) for steel rebars. To preserve the geometric relation, the ID of the four vertices on a face and the face ID of an element are documented. Each rebar is represented by several 3D truss elements with two nodes at their extremities. To aid in the graphic reconstruction of the rebar in Blender, the truss elements and the associated node ID for each rebar are systematically recorded. Duplicates are removed from the nodal coordinates.

In the Blender environment, concrete specimens are modeled face by face using corresponding vertex coordinates. Superfluous internal nodes and faces are eliminated to streamline the model and minimize computational time. Figure 13(a) and (b) show the concrete specimens in Abaqus and Blender, respectively. Moreover, rebar is rendered using a Bezier curve, which necessitates a sequence of nodal coordinates and computes the direction vectors at control points to achieve a smooth and continuous element. The reinforcement configurations in Abaqus and Blender are compared in Figure 13(c) and (d).

Figure 13.

Digital twin of shear wall model: (a) concrete specimen in Abaqus, (b) concrete specimen in Blender, (c) reinforcement in Abaqus, and (d) reinforcement in Blender.

Model texturing

The GBDT, derived from the FE model, employs a physically based rendering approach for texturing.⁴⁵ The light flow effect is simulated using several variables, namely albedo color, surface normal, roughness, and metallics.⁴⁶ Figure 14 showcases the influence of diverse types of texture variables. Within Blender, these variables are separately loaded in image format, referred to as texture maps, and combined via the Blender Principle Bidirectional Scattering Distribution Function (BSDF) node.

Figure 14.

Influence of texture variables.⁴⁶

Alongside the variables stated, displacement maps from the FE model are incorporated to form 3D concrete damage representation, such as cracks and spalling. The displacement map of concrete damage is derived from the FE models. The contour of the displacement map mirrors that of the damage map. The intensity of the displacement map, indicating the depth of damage at different locations, is determined by the total length of damaged elements within the shear wall’s depth. The displacement map facilitates 3D surface geometric modifications that interact with various structural effects, including exposed rebar, shadowing, and structural deformation of the concrete wall. A node group is utilized in Blender, enabling the efficient integration of texture maps into realistic textures.

Damage map texturing

In this study, concrete and reinforcement textures were sourced from online texture libraries.^47–49 Two sets of concrete textures were utilized to distinguish between concrete surfaces and internal cracks. All texture maps were deployed without post-processing, excluding the displacement map. To illustrate the surface displacement resultant from damage, the displacement map was manipulated based on the concrete damage map.

Displacement and normal maps within texture assets both represent surface unevenness. Displacement maps display substantial and global displacement variations on a surface by distorting meshes, constrained by the size and mesh density. In contrast, normal maps simulate the lighting of bumps and indents by defining the surface normal for pixels, thereby providing high-resolution lighting changes without mesh distortion. In this study, the damage displacement map, derived from the damage map, conveys the pronounced displacement changes due to damage, while the normal map from the texture assets remains unchanged to depict realistic material surfaces. Figure 15 provides an example of a damaged surface.

Figure 15.

Example of damaged surface.

Synthetic image rendering

Once the GBDT has been created, images with realistic damage are rendered using Blender’s inbuilt virtual camera. The camera is strategically positioned in front of the concrete wall, aimed perpendicularly at the front wall. The camera’s view is designed to encompass the entire configuration of the concrete wall specimen, mirroring the imaging approach used in the validation experiments.^32,36 Pairs of synthetic images are rendered and archived in the dataset for various parameters and used for subsequent forward and backward prediction training. A sample set of synthetic images captured from the damaged GBDT is displayed and compared against the image from the experiment in Figure 16. Noting that Figure 16 is only a showcase of the rendered GBDT, which aims at introducing and illustrating the complete forward production in the Bi-GBDT framework rather than creating a photo-realistic synthetic environment. Therefore, the details of the rendered GBDT image are not further polished.

Figure 16.

Sample set of rendered GBDT and experimental images: (a) rendered GBDT and (b) experimental image.

Backward prediction based on DL networks

This section seeks to predict the structural damage measures and conditions from identified damage images, which is termed backward prediction. This prediction utilizes two different DL networks, namely Residual Neural Network (ResNet) and Conditional Generative Adversarial Networks (cGANs).^50,51 These networks are trained on synthetic datasets, the results of which are generated from the damaged GBDT following forward prediction.

Importantly, the backward prediction is not restricted to a specific format of visual damage, whether they are unprocessed images from photographic surveys or identified damage maps. As introduced in section “Methodology framework,” damage identification has been well studied. Therefore, the focus of this section is on establishing the connection between structural conditions and identified damage. In addition, the workflow of the overall forward and backward prediction will be presented in section “Backward prediction based on DL networks.”

To investigate the performance of backward prediction on different scenarios, the structural damage parameters are expressed in two scales: (1) overall description, exemplified by maximum drift, and (2) refined parameters, such as the stress field and strain field. In the second scale, the prediction target is further differentiated into von Mises Stress, a simplified overview criterion, and horizontal and vertical strain, two main components for a detailed representation of deformation. To predict these parameters, ResNet and cGANs are modified and implemented, termed Damage2Drift (D2Drift) and Damage2Stress (D2Stress), respectively. The term “Stress” in Damage2Stress can be changed based on the specific network output, as such Damage2HStrain (D2HStrain) and Damage2VStrain (D2VStrain).

Damage2Drift

This section develops Damage2Drift, which is a modified Residual Neural Network (ResNet), to determine the maximum drift of the shear wall from the damage maps.⁵⁰ Subsequently, the drift predicted by D2Drift is classified into different damage states, adhering to the standard stipulated by the Federal Emergency Management Agency (FEMA).⁴²

Dataset

The dataset designed for D2Drift comprises a synthetic set for training, validation, and testing, as well as a physical dataset employed to gauge its stability against real-world damage inputs. From the experiment discussed in section “Seismically designed reinforced concrete shear wall,” a total of 18 damage maps with drifts below 0.5% are included in the physical dataset. As discussed before, drifts larger than 0.5% are not discussed in this study as they are readily identifiable.

The synthetic dataset is constructed from the diverse damage patterns simulated from 100 randomly generated loading histories, following the forward prediction process using proposed FE model and associated GBDT. Following the methodology deployed in the initial experiment, each loading history involves seven loading cycles. As represented in Figure 17(a), this pattern incorporates five distinct and sequentially increasing drift levels, with the initial two levels occurring twice. The drift is sampled from a uniform distribution corresponding to a drift ratio between 0.0% and 0.5% to encompass the broadest possible range of drift values. Table 3 displays the range for five drift levels.

Figure 17.

Introduction of the synthetic dataset: (a) example of a loading history and (b) histogram of drift values.

Table 3.

Range of sampled drift levels.

Drift level	Level 1	Level 2	Level 3	Level 4	Level 5
Drift (%)	0–0.10	0.10–0.15	0.15–0.25	0.25–0.35	0.35–0.50
Drift (mm)	0–3.66	3.66–5.49	5.49–9.14	9.14–12.80	12.80–18.29

To fully utilize the complete loading history, damage maps on all local extrema drift values are collected, resulting in the generation of 1400 damage maps and their corresponding damage parameters, drift. The histogram of drift values contained within the established dataset is displayed in Figure 17(b).

Then, an additional step filters out 310 damage maps associated with zero damage, as these null damage maps, despite their absence of damage, correspond to various distinct drift values. Finally, the remaining 1090 damage maps are allocated to training, validation, and test sets, containing 800, 90, and 200 maps, respectively.

Network

Initially proposed for image recognition, ResNet is well-recognized for its residual block design featuring “shortcut connections.” These connections foster accuracy enhancements from augmented network depth and have found extensive applications in subsequent networks for diverse tasks, such as semantic segmentation, depth estimation, and response time-series prediction.^52–54

The D2Drift architecture is comprised of 17 convolutional layers and a fully connected layer, processing inputs from 512 × 512 damage maps and yielding numerical drift values. Each convolutional layer is succeeded by a batch normalization layer and a Rectified Linear Unit (ReLU) activation function.^55,56 A shortcut connection is integrated after every two convolutional layers, forming a residual block. The network ends with a flattened layer and a fully connected layer. The architecture of D2Drift is visually represented in Figure 18, with layer specifics provided in Table 4.

Figure 18.

Illustration of architecture of D2Drift: (a) overall architecture and (b) structure of residual block.

Table 4.

Details of D2Drift. Residual blocks are shown in brackets with filter size and number of channels.

Layer name	Output size	Convolutional layer
Conv 1	$256 \times 256$	$7 \times 7$ , 64, stride 2
Max Pool 1	$128 \times 128$	$3 \times 3$ , 64, stride 2
Residual Block 1	$128 \times 128$	$3 \times 3, 64$
Residual Block 2	$128 \times 128$	$3 \times 3, 64$
Residual Block 3	$64 \times 64$	$3 \times 3, 128$ , stride 2
Residual Block 4	$64 \times 64$	$3 \times 3, 128$
Residual Block 5	$32 \times 32$	$3 \times 3, 256$ , stride 2
Residual Block 6	$32 \times 32$	$3 \times 3, 256$
Residual Block 7	$16 \times 16$	$3 \times 3, 256$ , stride 2
Residual Block 8	$16 \times 16$	$3 \times 3, 256$
	$1 \times 1$	Flatten, fully connected

D2Drift: Damage2Drift.

Training of the D2Drift model occurs on the synthetic dataset, processing batches of five, and utilizes the Adam optimization algorithm.⁵⁷ The “Mean Squared Error (MSE)” loss function is employed, with pre-specified learning rates set at 1 × 10⁻³, 1 × 10⁻⁴, and 1 × 10⁻⁵ for 100, 50, and 100 epochs, respectively.

Results and deviation analysis

The performance of the D2Drift network is evaluated through three metrics: (1) MSE, (2) Mean Absolute Error (MAE), and (3) Mean Absolute Percentage Error (MAPE). These metrics give a thorough evaluation, with MAE being general, MSE sensitive to outliers, and MAPE clear in percentage-based errors.

$\begin{matrix} MSE = \frac{\sum_{i = 1}^{n} {(y_{i} - {\hat{y}}_{i})}^{2}}{n}, MAE = \frac{\sum_{i = 1}^{n} | y_{i} - {\hat{y}}_{i} |}{n}, \\ MAPE = \frac{100}{n} \sum_{i = 1}^{n} | \frac{y_{i} - {\hat{y}}_{i}}{{\hat{y}}_{i}} | \end{matrix}$ (6)

where ${\hat{y}}_{i}$ represents the ground truth drift, and $y_{i}$ denotes the predicted drift.

An initial evaluation employs a synthetic test set comprising 200 data pairs simulated using the proposed shear-wall model. As indicated in Table 5, the D2Drift network exhibits accurate predictions for the synthetic test set, with an average deviation of 5.10% (0.42 mm). This performance verifies its efficacy in predicting maximum drift based on damage maps.

Table 5.

Evaluation of D2Drift.

Metric	Synthetic			Physical
	Training	Validation	Test	Test
MSE (mm²)	0.087	0.30	0.35	1.77
MAE (mm)	0.25	0.38	0.42	1.01
MAPE	3.39%	4.84%	5.10%	10.41%

D2Drift: Damage2Drift; MSE: mean squared error; MAE: mean absolute error; MAPE: mean absolute percentage error.

Subsequent testing on a physical dataset further demonstrates the robustness of the D2Drift network when dealing with new data featuring physical damage patterns. Despite being trained solely on a synthetic dataset generated through FE simulation, the D2Drift network consistently delivers accurate drift predictions with an average deviation of 10.41% (1.01 mm).

Figure 19 illustrates a comparative representation of the predicted drift and corresponding ground truth across two test sets. D2Drift demonstrates accurate and stable performance on both datasets, particularly for drift values below 14 mm. The maximum error, a 2.75 mm discrepancy, is observed in the case highlighted in Figure 19. In this instance, the predicted drift deviates by 19.56% from the actual drift, which is an acceptable deviation considering it is the largest drift level within the physical dataset.

Figure 19.

Predicted drift versus actual drift of synthetic and physical test sets.

To further study the results and the impact of the dataset, a comparison of synthetic and physical data at different drift levels is illustrated in each row of Figure 20. Three key drift levels identifiable from the physical data, 3.87, 10.75, and 14.03 mm, are selected for this purpose. For each of these drift levels, synthetic data with closest drift value from distinct loading histories are selected from both the training and test sets.

Figure 20.

Examples of maximum drift prediction and comparison with experiments.

At a drift value of approximately 3.88 mm, the synthetic case exhibits less damage than the physical case, leading to an overestimation of maximum drifts on the physical damage map. At the drift value of around 10 mm, the synthetic data accurately mirrors the physical data’s damage pattern, as corroborated by comparing distribution, direction, and density of damage. Hence, the error at this drift level is lower than the overall MAPE outlined in Table 5. At a drift value of around 14 mm, dense damage is seen at the lower level of the synthetic damage map. In contrast, the physical damage map exhibits minimal alteration from its 10 mm version, causing an underestimation in predicted maximum drift.

In summary, the accuracy of predictions is largely contingent upon the precision of the synthetic damage relative to physical damage. The performance of the D2Drift network is good and could be further enhanced by obtaining more accurate simulation results.

Damage states estimation

Various standards and guidelines have been developed to estimate the structural damage states based on maximum inter-story drift levels. The American Society of Civil Engineers (ASCE) emphasizes the importance of the maximum inter-story drift ratio as reliable metrics for rapid post-seismic structural condition assessment.⁵⁸ The FEMA divided structural damage into one of four states: “Slight,”“Moderate,”“Extensive,” or “Complete.”⁴² For concrete shear walls in high-rise buildings, a drift ratio within [0.2%, 0.5%] is specifically categorized as “Slight” damage, while a drift ratio less than 0.2%, the damage is classified as “None.” Noting that, as discussed in section “Seismically designed reinforced concrete shear wall,” drift ratio larger than 0.5% is not considered in this study. The maximum drift predicted by D2Drift for the shear wall considered will be classified as either safe or slightly damaged.

The effectiveness of the proposed method is evaluated using metrics, such as recall, precision, and F1 score, with results visualized in the confusion matrix (Figure 21). As shown in Table 6 and Figure 21, the proposed method estimates structural damage states accurately for both synthetic and experimental data, reinforcing the robustness of the introduced Bi-GBDT framework. Crucially, the meaning of the visual damage can be precisely inferred after the detection and identification, which bridges the gap by focusing on accurate visual damage prediction in both forward and backward directions derived from GBDT.

Figure 21.

Confusion matrix of predicted damage states: (a) training, (b) validation, (c) test and (d) physical test.

Table 6.

Evaluation of predicted damage states.

Metric	Synthetic			Physical
	Training	Validation	Test	Test
Accuracy	99.50%	97.78%	98.50%	100%
Precision	1.00	0.97	0.98	1.00
Recall	0.99	0.97	0.99	1.00
F1 score	0.99	0.97	0.98	1.00

D2Stress, D2HStrain, and D2VStrain

The prediction of graphic parameters is accomplished using a set of customized cGANs.⁵¹ These networks are termed Damage2Stress (D2Stress), Damage2HStrain (D2HStrain), and Damage2VStrain (D2VStrain), corresponding to its output for stress field, horizontal strain field, and vertical strain field, respectively. Although the inter-story drift ratio discussed above reflect the overall situations of the structure, predicting the stress and strain fields offer the measures in a more refined scale to localize the damaged region on a structure and enables multi-scale structural damage assessment.^59,60

Dataset

D2Stress, D2HStrain, and D2VStrain are employed in three predictive tasks, conversion from damage map to stress field (i.e., von Mises Stress), horizontal strain field, and vertical strain field, respectively. In accordance with these separate tasks, three distinct synthetic datasets are assembled. These datasets stem from the identical simulation results utilized in the D2Drift model, as described in section “Damage2Drift.”

Similar to the approach adopted with D2Drift, the synthetic datasets pertaining to D2Stress, D2HStrain, and D2VStrain comprise 800, 90, and 200 paired data units of damage map and stress or strain fields for training, validation, and testing, respectively. Nevertheless, it must be noted that stress and strain fields are not accessible from the experiments. Therefore, the performance evaluations of D2Stress, D2HStrain, and D2VStrain are executed solely on the synthetic test set.

Network

The identical network architecture is utilized in this study for three predictive tasks, thereby demonstrating the efficacy and successful implementation of the backward prediction of Bi-GBDT. A modified version of cGANs offers translation capabilities from image to image between input damage maps and output stress or strain fields. Generative adversarial networks (GANs), the blueprint of cGANs, simultaneously train a generator and a discriminator.⁶¹ While the generator fabricates images and attempts to deceive the discriminator with these synthesized images, the discriminator is simultaneously trained to detect such counterfeits. GANs can learn to map from random noise inputs to target output images. This key functionality of GANs is retained and adapted in cGANs for conditional translation from input images to target output images.

The architecture of the generator is predicated on the U-Net model, which is widely recognized in image generation tasks.⁶² The generator employs eight convolutional layers for feature map extraction, each succeeded by a batch normalization and a leaky ReLU activation function. The feature map is subsequently unsampled by eight deconvolutional layers, each followed by batch normalization and ReLU activation. Furthermore, the discriminator model is a singular encoder structure that refines the features of both input damage map and generated strain or stress field into a 16 × 16 patch of probabilities that represent the authenticity of the generated field. Figure 22 illustrates the architecture of D2Stress, D2HStrain, and D2VStrain. Meanwhile, detailed descriptions of each layer can be found in Tables 7 and 8.

Figure 22.

Illustration of architecture of D2Stress, D2HStrain, and D2VStrain: (a) generator and (b) discriminator.

Table 7.

Layer details of generative model.

Layer name	Output size	Convolutional layer
conv1 & deconv7	$128 \times 128$	$4 \times 4$ , 64, stride 2
conv2 & deconv6	$64 \times 64$	$4 \times 4, 128, stride 2$
conv3 & deconv5	$32 \times 32$	$4 \times 4, 256, stride 2$
conv4 & deconv4	$16 \times 16$	$4 \times 4, 512, stride 2$
conv5 & deconv3	$8 \times 8$	$4 \times 4, 512, stride 2$
conv6 & deconv2	$4 \times 4$	$4 \times 4, 512, stride 2$
conv7 & deconv1	$2 \times 2$	$4 \times 4, 512, stride 2$
conv8	$1 \times 1$	$4 \times 4, 512, stride 2$
deconv8	$256 \times 256$	$4 \times 4, 1, stride 2$

Table 8.

Layer details of discriminative model.

Layer name	Output size	Convolutional layer
conv1	$128 \times 128$	$4 \times 4$ , 64, stride 2
conv2	$64 \times 64$	$4 \times 4, 128, stride 2$
conv3	$32 \times 32$	$4 \times 4, 256, stride 2$
conv4	$16 \times 16$	$4 \times 4, 512, stride 2$
conv5	$16 \times 16$	$4 \times 4, 512$
conv6	$16 \times 16$	$4 \times 4, 1$

The training of three models of D2Stress, D2HStrain, and D2VStrain involve a learning rate of 1 × 10⁻³, 1 × 10⁻⁴, and 1 × 10⁻⁵ for 50, 10, and 90 epochs, respectively, after comparison. The model employs the Adam optimizer with a batch size of one, following the recommendations in the cGANs paper.⁵¹

Results

The efficacy of D2Stress, D2HStrain, and D2VStrain in predicting stress or strain field is measured via two metrics: MSE and MAE. These metrics are computed using 200 synthetic test data pairs and 18 simulated physical outcomes from the experiment, given the unavailability of physical data.

As detailed in Table 9, D2Stress, D2HStrain, and D2VStrain achieve accurate predictions for stress, horizontal strain field, and vertical strain field. Specifically, D2Stress yields an average error of 0.95 MPa on the test set and 1.95 MPa on the experimental test set, relative to a real stress range of 0–68 MPa. Moreover, D2HStrain and D2VStrain report average errors of 9.99 × 10⁻⁵ and 3.17 × 10⁻⁴ on the test set, and 4.16 × 10⁻⁴ and 1.70 × 10⁻³ on experimental test set, respectively.

Table 9.

Evaluation of D2Stress, D2HStrain, and D2VStrain.

Target	Metric	Synthetic
		Training	Validation	Test	Experiment
D2Stress	MSE (MPa²)	1.42	2.36	3.96	8.87
	MAE (MPa)	0.82	0.88	0.95	1.95
D2HStrain	MSE	7.98 × 10⁻⁹	9.01 × 10⁻⁸	1.29 × 10⁻⁷	1.41 × 10⁻⁶
	MAE	4.61 × 10⁻⁵	7.60 × 10⁻⁵	9.99 × 10⁻⁵	4.16 × 10⁻⁴
D2VStrain	MSE	8.69 × 10⁸	2.55 × 10⁻⁶	1.69 × 10⁻⁶	3.11 × 10⁻⁵
	MAE	1.62 × 10⁻⁴	3.62 × 10⁻⁴	3.17 × 10⁻⁴	1.70 × 10⁻³

D2Stress: Damage2Stress; D2HStrain: Damage2HStrain; D2VStrain: Damage2VStrain; MSE: mean squared error; MAE: mean absolute error.

Figure 23 showcases examples of the predictive outcomes of D2Stress, D2HStrain, and D2VStrain. The accurate predictions for stress field, along with horizontal and vertical strain fields, demonstrate the effectiveness of the proposed method. Furthermore, certain damage patterns can be discerned across all simulated stress and strain fields, as the mesh at such locations exhibits significantly degraded stiffness, resulting in significantly different strains and stresses compared to the undamaged areas.

Figure 23.

Examples of stress and strain prediction.

Backward prediction from rendered image

After completing the forward prediction workflow, the rendered images that simulate the photographic survey can be used as input for backward prediction. As introduced in section “Methodology framework,” backward prediction consists of two main steps. First, the damage in the image is identified using damage detection algorithms. Subsequently, the damage states and measures can be predicted by DL networks based on such damage maps. This subsection showcases both steps of the backward prediction from rendered images to the damage measures and structural conditions, which is not discussed in section “Generation of GBDT using forward prediction” directly.

Figure 24 presents an image rendered from damaged GBDT and the corresponding identified damage map using the widely accepted computer vision-based detection approach.⁶³ Then, the damage measure and damage state are predicted following the method described in section “Geometric information.” As shown in Table 10, the backward prediction performs well in predicting drift with a deviation of 5.46%. Moreover, the agreement between the predicted and true damage states serves to demonstrate the reliability and effectiveness of the Bi-GBDT framework as a whole.

Figure 24.

Example of damage detection: (a) rendered image and (b) identified damage map.

Table 10.

Evaluation of backward prediction results.

Result	Prediction	Ground truth
Drift	7.76 mm	8.21 mm
Damage state	“None”	“None”

Conclusion

This study proposes a general framework, termed Bi-GBDT, to establish the connection between structural conditions and identified damage patterns from post-earthquake UAV photographic surveys. The forward prediction that generates accurate damage patterns from proposed FE modeling and simulation is validated against real-world experiments on a concrete shear wall. The backward prediction of multi-scale structural condition assessment using DL networks is also verified for its effectiveness and robustness in this study. The following conclusions could be drawn:

The proposed FE modeling and simulation methods present an effective representation of physical damage observed in a concrete shear wall subjected to cyclic load. The simulation successfully mimics the experimental damage patterns considering the hysteretic behavior and distribution of damage for drifts below 0.5%.

Based on the proposed FE simulation and graphics modeling approach, the GBDT of the structure can be established by the accurate and validated damage pattern exemplified by the concrete shear wall model. The GBDT is able to generate synthetic images based on the demand of researchers. In this study, the synthetic images are used to train the DL network for backward prediction in subsequent sections.

The backward prediction of the damage measures (i.e., maximum drift, stress and strain field) and structural condition are achieved by modified Residual Neural Network and Conditional GANs. The effectiveness and robustness of the networks are comprehensively validated against experimental and synthetic data. The success of backward prediction offers valuable references for accurate digital twins and automated post-earthquake assessment from photographic surveys.

The application of backward prediction on rendered images demonstrates the robustness and effectiveness of the Bi-GBDT framework in estimating structural conditions using images from photographic surveys.

Drawing upon its capacity to accurately forecast validated structural damage metrics (including maximum drift, stress, and strain fields) from visible damage, the proposed Bi-GBDT bridges the gap between visual phenomena and multi-scale damage states. This research establishes a substantive link between the digital twin and the physical structure. Distinct from earlier studies that merely modeled and identified visible damage, Bi-GBDT offers profound implications, marking a considerable advancement toward the realization of a comprehensive digital twin.

Footnotes

The authors would like to express their sincere thanks to Professor Hui Li,Professor Jinhui Yan,Professor Jiazeng Shan,Professor Shurong Li,and PhD Candidate Shuo Wang for providing valuable suggestions.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research,authorship,and/or publication of this article: In addition,the second author was supported in part by Young Scientists Fund from the National Natural Science Foundation of China under grants No. 52308536. The first author was supported in part by the China Scholarship Council under grants No. 201908040012. This work is partially supported by the National Natural Science Foundation of China (No. 51978182).

ORCID iD

Guanghao Zhai

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request. The networks utilized in this paper are available for public access and can be found in the GitHub repository at

after the publication of this study.

References

Applied Technology Council (ATC). ATC-20: Procedures for postearthquake safety evaluation of buildings. Redwood, CA: Applied Technology Council, 1989.

Dandoulaki

Panoutsopoulou

Ioannides

. An overview of post-earthquake building inspection practices in Greece and the introduction of a rapid building usability evaluation procedure after the 1996 Konitsa earthquake. In: Proceedings of 11th European conference on earthquake engineering, Paris, France, 1998.

Farrar

Worden

. An introduction to structural health monitoring. Philos Trans R Soc Math Phys Eng Sci 2006; 365: 303–315.

Lozano

J-M

Tien

. Data collection tools for post-disaster damage assessment of building and lifeline infrastructure systems. Int J Disaster Risk Reduct 2023; 94: 103819.

Zhou

Tang

, et al. A review of vision-laser-based civil infrastructure inspection and monitoring. Sensors 2022; 22: 5882.

Morgenthal

Hallermann

. Quality assessment of unmanned aerial vehicle (UAV) based visual inspection of structures. Adv Struct Eng 2014; 17: 289–302.

Nedjati

Izbirak

Vizvari

, et al. Complete coverage path planning for a multi-UAV response system in post-earthquake assessment. Robotics 2016; 5: 26.

Waqas

Kang

Cha

Y-J

. Deep learning-based obstacle-avoiding autonomous UAVs with fiducial marker-based localization for structural health monitoring. Struct Health Monit. Epub ahead of print 21 June 2023. DOI: 10.1177/14759217231177314.

Bolourian

Hammad

. LiDAR-eq7 UAV path planning considering potential locations of defects for bridge inspection. Autom Constr 2020; 117: 103250.

10.

Phung

Quach

Dinh

, et al. Enhanced discrete particle swarm optimization path planning for UAV vision-based surface inspection. Autom Constr 2017; 81: 25–33.

11.

Bae

Jang

Y-K

. Deep super resolution crack network (SrcNet) for improving computer vision–based automated crack detectability in in situ bridges. Struct Health Monit 2021; 20: 1428–1442.

12.

Jang

Kim

Y-K

. Deep learning–based autonomous concrete crack evaluation through hybrid image scanning. Struct Health Monit 2019; 18: 1722–1737.

13.

Zhang

Wang

KCP

, et al. Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network. Comput-Aided Civ Infrastruct Eng 2017; 32: 805–819.

14.

Ghosh Mondal

Jahanshahi

R-T

, et al. Deep learning-based multi-class damage detection for autonomous post-disaster reconnaissance. Struct Control Health Monit 2020; 27: e2507.

15.

Kim

Cho

. Automated multiple concrete damage detection using instance segmentation deep learning model. Appl Sci 2020; 10: 8008.

16.

Zou

Zhang

Bai

, et al. Multicategory damage detection and safety assessment of post-earthquake reinforced concrete structures using deep learning. Comput-Aided Civ Infrastruct Eng 2022; 37: 1188–1204.

17.

Spencer

Hoskere

Narazaki

. Advances in computer vision-based civil infrastructure inspection and monitoring. Engineering 2019; 5: 199–222.

18.

Hoskere

Narazaki

Hoang

, et al. Towards Automated post-earthquake inspections with deep learning-based condition-aware models. Epub ahead of print 2018. DOI: 10.48550/arXiv.1809.09195.

19.

Hoskere

Narazaki

Spencer

. Learning to detect important visual changes for structural inspections using physicsbased graphics models: 9th International Conference on Structural Health Monitoring of Intelligent Infrastructure: Transferring Research into Practice, SHMII 2019. In: Chen

Alampalli

(eds) 9th International Conference on Structural Health Monitoring of Intelligent Infrastructure, St. Louis, Mo. International Society for Structural Health Monitoring of Intelligent Infrastructure, ISHMII, pp. 1484–1490.

20.

Hoskere

Narazaki

Spencer

. Physics-based graphics models in 3D synthetic environments as autonomous vision-based inspection testbeds. Sensors 2022; 22: 532.

21.

Narazaki

Hoskere

Yoshida

, et al. Synthetic environments for vision-based structural condition assessment of Japanese high-speed railway viaducts. Mech Syst Signal Process 2021; 160: 107850.

22.

Narazaki

Hoskere

Chowdhary

, et al. Vision-based navigation planning for autonomous post-earthquake inspection of reinforced concrete railway viaducts using unmanned aerial vehicles. Autom Constr 2022; 137: 104214.

23.

Wang

Rodgers

Zhai

, et al. A graphics-based digital twin framework for computer vision-based post-earthquake structural inspection and evaluation using unmanned aerial vehicles. J Infrastruct Intell Resil 2022; 1: 100003.

24.

Levine

Narazaki

Spencer

. Performance-based post-earthquake building evaluations using computer vision-derived damage observations. Adv Struct Eng 2022; 25: 3425–3449.

25.

Agrawal

Washer

Gong

, et al. Consistency of the New York State bridge inspection program. C-07–17, New York (NY). Dept. of Transportation, https://rosap.ntl.bts.gov/view/dot/26549 (2013, accessed 22 September 2023).

26.

Avci

Abdeljaber

Kiranyaz

, et al. A review of vibration-based damage detection in civil structures: from traditional methods to machine learning and deep learning applications. Mech Syst Signal Process 2021; 147: 107077.

27.

Moehle

Deierlein

. A framework methodology for performance-based earthquake engineering. In: 13th World conference on earthquake engineering. Vancouver: WCEE, 2004, p. 12.

28.

Bruneau

Reinhorn

. Exploring the concept of seismic resilience for acute care facilities. Earthq Spectra 2007; 23: 41–62.

29.

Ebrahimkhanlou

Farhidzadeh

Salamone

. Multifractal analysis of crack patterns in reinforced concrete shear walls. Struct Health Monit 2016; 15: 81–92.

30.

Bazrafshan

Basereh

, et al. A graph-based method for quantifying crack patterns on reinforced concrete shear walls. Comput-Aided Civ Infrastruct Eng. Epub ahead of print 19 April 2023. DOI: 10.1111/mice.13009.

31.

Raissi

Perdikaris

Karniadakis

. Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J Comput Phys 2019; 378: 686–707.

32.

Lowes

Lehman

Birely

, et al. Earthquake response of slender planar concrete walls with modern detailing. Eng Struct 2012; 43: 31–47.

33.

Prasanna

Dana

Gucunski

, et al. Automated crack detection on concrete bridges. IEEE Trans Autom Sci Eng 2016; 13: 591–599.

34.

Yeum

Dyke

. Vision-based automated crack detection for bridge inspection. Comput-Aided Civ Infrastruct Eng 2015; 30: 759–770.

35.

Dassault Systèmes. ABAQUS, https://www.3ds.com/products-services/simulia/products/abaqus/ (2020, accessed 31 July 2022).

36.

Birely

. Seismic Performance of Slender Reinforced Concrete Structural Walls. PhD Thesis, University of Washington, https://www.proquest.com/docview/1313210383/abstract/48390B7EA0584B0FPQ/1 (2012, accessed 25 April 2022).

37.

Hognestad

. Study of combined bending and axial load in reinforced concrete members. 399, University of Illinois. Engineering Experiment Station, https://www.ideals.illinois.edu/handle/2142/4360 (1951, accessed 25 April 2022).

38.

Cornelissen

HaW

Hordijk

Reinhardt

. Experimental determination of crack softening characteristics of normalweight and lightweight concrete. HERON 31 2 1986, https://repository.tudelft.nl/islandora/object/uuid%3A08c29b39-5c60-4ab6-b9d5-643d11007f7c (1986, accessed 25 April 2022).

39.

Lee

Fenves

. Plastic-damage model for cyclic loading of concrete structures. J Eng Mech 1998; 124: 892–900.

40.

Lubliner

Oliver

Oller

, et al. A plastic-damage model for concrete. Int J Solids Struct 1989; 25: 299–326.

41.

Hafezolghorani

Hejazi

Vaghei

, et al. Simplified damage plasticity model for concrete. Struct Eng Int 2017; 27: 68–78.

42.

Federal Emergency Management Agency (FEMA). In: HAZUS-MH 2.1 earthquake model technical manual. Washington, D.C., USA: Federal Emergency Management Agency (FEMA), 2013.

43.

Blender Foundation. Blender, https://www.blender.org/ (2023, accessed 28 October 2023).

44.

Perlin

. An image synthesizer. ACM SIGGRAPH Comput Graph 1985; 19: 287–296.

45.

Burley

Walt

Disney Animation Studios. Physically based shading at disney. ACM SIGGRAPH Comput Graph 2012; 2012: 1–7.

46.

Zhai

Narazaki

Wang

, et al. Synthetic data augmentation for pixel-wise steel fatigue crack identification using fully convolutional networks. Smart Struct Syst 2022; 29: 237–250.

47.

ambientCG. ambientCG, https://ambientCG.com/ (2023, accessed 21 June 2023).

48.

Poliigon. Poliigon, https://www.poliigon.com (2023, accessed 21 June 2023).

49.

Textures.com. Textures.com, https://www.textures.com/ (2023, accessed 21 June 2023).

50.

Zhang

Ren

, et al. Deep residual learning for image recognition. Epub ahead of print 10 December 2015. DOI: 10.48550/arXiv.1512.03385.

51.

Isola

Zhu

J-Y

Zhou

, et al. Image-to-image translation with conditional adversarial networks. Honolulu, HI: IEEE, 2017, pp. 1125–1134.

52.

Koltun

Funkhouser

. Dilated residual networks. Honolulu, HI: IEEE, 2017, pp. 472–480.

53.

Laina

Rupprecht

Belagiannis

, et al. Deeper depth prediction with fully convolutional residual networks. In: 2016 Fourth international conference on 3D vision (3DV), Stanford, CA, 2016, pp. 239–248.

54.

Fei

Huang

, et al. Advanced corrective training strategy for surrogating complex hysteretic behavior. Structures 2022; 41: 1792–1803.

55.

Ioffe

Szegedy

. Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd international conference on machine learning, Lille, France. PMLR, pp. 448–456.

56.

Nair

Hinton

. Rectified linear units improve restricted boltzmann machines, https://openreview.net/forum?id=rkb15iZdZB (2019, accessed 28 June 2022).

57.

Kingma

. Adam: a method for stochastic optimization. Epub ahead of print 29 January 2017. DOI: 10.48550/arXiv.1412.6980.

58.

American Society of Civil Engineers (ASCE). Minimum design loads for buildings and other structures. 7th–10th ed. Reston, VA: ASCE/SEI, 2010.

59.

Sohn

Farrar

Hemez

, et al. A review of structural health review of structural health monitoring literature 1996–2001. LA-UR-02-2095, Los Alamos National Lab. (LANL), Los Alamos, NM, USA, https://www.osti.gov/biblio/976152 (2002, accessed 17 October 2023).

60.

Sutton

Orteu

Schreier

. Image correlation for shape, motion and deformation measurements: basic concepts, theory and applications. New York, NY: Springer Science & Business Media, 2009.

61.

Goodfellow

Pouget-Abadie

Mirza

, et al. Generative adversarial nets. In: Ghahramani

Welling

Cortes

, et al. (eds) Advances in neural information processing systems. Curran Associates, Inc., https://proceedings.neurips.cc/paper/2014/hash/5ca3e9b122f61f8f06494c97b1afccf3-Abstract.html (2014, accessed 28 June 2022).

62.

Ronneberger

Fischer

Brox

. U-Net: convolutional networks for biomedical image segmentation. In: Navab

Hornegger

Wells

, et al. (eds) Medical image computing and computer-assisted intervention – MICCAI 2015. Cham: Springer International Publishing, 2015, pp. 234–241.

63.

Zou

Cao

, et al. CrackTree: automatic crack detection from pavement images. Pattern Recognit Lett 2012; 33: 227–238.

Bidirectional graphics-based digital twin framework for quantifying seismic damage of structures using deep learning networks

Abstract

Keywords

Introduction

Methodology framework

Seismically designed reinforced concrete shear wall

Experiment setup

FE model configuration

Constitutive model

Concrete

Steel

Comparison with experimental data

Load–displacement response

Damage patterns and failure modes

Generation of GBDT using forward prediction

Geometric information

Model texturing

Damage map texturing

Synthetic image rendering

Backward prediction based on DL networks

Damage2Drift

Dataset

Network

Results and deviation analysis

Damage states estimation

D2Stress, D2HStrain, and D2VStrain

Dataset

Network

Results

Backward prediction from rendered image

Conclusion

Footnotes

Declaration of conflicting interests

Funding

ORCID iD

Data availability

References