Source: https://opg.optica.org/ol/fulltext.cfm?uri=ol-46-17-4212
Interest in nonglasses three-dimensional (3D) displays over the past decades has led to the development of light field displays [1–8]. The light field display can modulate both the direction and intensity of light, which provides the capability to reconstruct 3D objects in free space. However, it is inherently restricted by the information capacity of a flat panel display, which is defined as the number of pixels. Typically, to represent a single point, a bundle of converging-diverging rays should be used. Since the ray and the pixel are one-to-one matched, it reduces the available information by its number. This is a fundamental trade-off between spatial and angular resolution.
A critical issue in the light field displays is the need for enormous information to achieve enough visual quality, full-parallax, and depth. It may be the prime reason why the light field displays are difficult to put into practical use. To increase the information capacity and extend the 3D volume, the spatial-multiplexed systems of stacking multiple two-dimensional (2D) displays in the depth direction were proposed [1,2]. Furthermore, approaches that computationally optimize the light field have been described, which can condense the spatial-angular information effectively [3,4]. However, such a stacking structure has a problem that a bulky space and a large number of display panels are essential to be implemented on a large scale.
In contrast, the projection-based light field system has the advantage of being able to scale the display at will [5,6]. Using multiple projectors can easily increase the information capacity [7,8]. Integral imaging (InIm) is the most straightforward and suitable method for a projection-based light field. In the InIm, the angular information is recorded and reproduced in the form of spatial information called an elemental image (EI), as shown in Fig.
1
(a). The problem is that the amount of information is limited by the spatial resolution of the pickup sensor or display device. Moreover, redundant information occurs to express the angular information of a 3D scene, as shown in Fig.
1
(b). Thereby, the InIm shows a significantly lower image quality than the original resolution of the display panel. We think that there exists a capability to improve the performance of the InIm projection by removing the repeated and inefficiently used information.
Fig. 1.
(a) Basic two processes of the InIm system: pickup and reconstruction. (b) Number of picked up points is increasing according to distance. (c) Simplified concept diagram of the optical structure. It consists of four steps, optically combining multifocal display with integral imaging. Note that the can be a negative value, which means pickup as a virtual image.
Download Full
Size | PDF
Here, we introduce a novel projection-type light field display that can effectively transfer spatial-angular information on a large screen. Figure
1
(c) shows the principle of the light field optical transmission, which consists of four steps: light field generation, pickup, projection, and reconstruction. The main idea is to optically connect all the processes using a projection system. With this configuration, the automatically mapped EI plays a key role in avoiding the inefficient use of information represented by the trade-off relationship in the InIm. Furthermore, unlike the previous InIm systems, the proposed design prevents hardware-related information reduction at the capture and display stage and allows the EI to be supersampled without discrete division.
Fig. 2.
Ray-tracing results of the proposed method. (a)–(f) Two-dimensional light field simulation results for a single plane. The horizontal axis represents the -direction position of the light field, and the vertical axis represents the tangent value of its angle. The light field emanating from a pixel has the same color. The center pixel is marked in black to trace the shape of the light field. Illustrations of the changes in spot size during (g) pickup and (h) reconstruction process. (i) Maximum resolution of the system corresponding to the Rayleigh criterion when a wavelength is 550 nm. Our prototype has a pickup area of with .
Download Full
Size | PDF
For the first step, we adopt the tomographic display to generate a light field. This method can produce a volumetric scene over a wide depth range by creating dozens of planes placed at different depths [9–12]. The multifocal planes (MFPs) are generated from a red/green/blue/depth (RGB-D) image by synchronizing a binary backlight with a focus-tunable lens (FTL). Here, we utilize the FTL as an aperture stop for the MFPs. While the FTL controls the floating position to , each focal plane has the same divergence angle and size with the telecentric relay [13]. Then, in the next step, the synthesized light field from the MFPs is picked up by microlens array (MiLA). In the , the overlap between each lenslet can be alleviated by matching the -number () of the MFPs to that of MiLA. Third, through the projection lens, the is enlarged by the magnification factor on the plane. We place the screen here. However, similar to previous projection InIm techniques, it is possible to use either a screen or direct projection method here. The difference between the two methods has been well described using parameters such as fill factor and depth range [14,15]. As the final step, the light field after the screen is reconstructed as it passes through the macrolens array (MaLA) in the reverse process of the pickup.
Generally, bringing the volumetric scene to the big screen is difficult because the wider the area during magnification, the narrower the angle of each display pixel [16]. However, we utilize the InIm techniques while projecting the volumetric scene. It allows that the angular information of the volumetric scene is converted into spatial information during the pickup process. Accordingly, even if the divergence angle for each pixel is reduced when projected, it is restored while the EI is returned back to the angular information in the reconstruction process. In other words, our approach not only avoids the information loss in the InIm but also effectively solves the enlargement problem of the volumetric scene.
Figures
2
(a)–
2
(f) show the light field analysis of the proposed system for a focal plane. We only count a single dimension for simplicity and analyze the light field as an ordered pair of the position and angle. First, the light field has a rectangular shape with a spatial length of and a tangent value of , where is the of MiLA. After the propagation to the MiLA, the light field is transformed into a parallelogram, as shown in Fig.
2
(b). Then, the light field is divided spatially by the interval , MiLA pitch, and the maximum tangent value is doubled. Here, the MiLA optically arranges the pixels in the plane according to the spatial position of each lenslet. At the screen plane, the light field’s divergence angle, which decreases during the magnification, is expanded again by diffusing. Since the screen is placed at the focal length of the MaLA in the focused mode [17], the light field after the MaLA has a rectangular shape as shown in the inset of Fig.
2
(e). Then, after propagating distance , the MaLA reproduces the focal plane where each display pixel appears as a sampled form, as shown in Fig.
2
(f). The distance is calculated as from the geometric relationship. Because the volumetric depth is proportional to and the viewing angle is proportional to , these two parameters can be customized by adjusting the between the pickup and the reconstruction.
We evaluate the system performance by deriving a point size at the reconstruction plane based on ray optics. As shown in Fig.
2
(g), the blur spot size at the plane is calculated as while picking up a point at distance. After projecting and scattering, the blur spot reconstructs the point at distance, as shown in Fig.
2
(h). The lateral size of the reconstruction point is . By substituting and , the reconstructed point size is calculated as a constant value of , regardless of the pickup distance . However, the blur spot size cannot be smaller than the diffraction limit for the MiLA in the pickup process. Considering the diffraction effect, the minimum spot formed by the plane wave is . In accordance with the Rayleigh criterion, the maximum spatial resolution of the system can be considered as , where is the pickup area. By increasing a numerical aperture of the MiLA and the pickup area, our optical design enables a large-scale volumetric display equivalent to the InIm of megapixels or higher, as shown in Fig.
2
(i). For instance, in the case of using a typical projection lens with the pickup area of and , the volumetric display with a resolution of up to 1.3 gigapixels is feasible. For the experiment, due to a lack of suitable off-the-shelf MaLA, the pickup area of the prototype was set to .
We implemented the light field generation system using a digital micromirror device (DMD, DLP9500) from Texas Instruments as the binary backlight, which has a full high-definition (FHD) resolution and 16 kHz operation speed with size. Due to the DMD mirror characteristic, which rotates at an angle of 45°, we only use a resolution of as the modulation area. The backlight image projected from the DMD is relayed to the transparent LCD with a magnification of using camera lenses. The LCD model used is Sharp LS029B3SX, and the effective resolution is (0.22 megapixels).
For the focus-tunable lens, we selected the EL10-30-TC of Optotune, which has a 10 mm aperture. The MiLA of the RPC photonics was used at a -number of 4 and a pitch of 100 µm. The focal length of is set to 40 mm to match the between the MFPs and MiLA. For the synchronization of the FTL and DMD, the data acquisition (DAQ) board from National Instrument is utilized. In the experiments, we generated 60 planes for the volume of with the real-time operation. The effective number of MiLA is . Because of the limitation of the experimental space, the magnification of is set to 16. However, the system scale can be expanded without restrictions. The MaLA has a pitch of 1.6 mm with . With this configuration, the system reconstructs the volumetric scene of . More system details are discussed in Supplement 1.
Figure
3
(a) shows the cropped for varying pickup distance from to 8 mm. As the is matched, the EI is confined within a boundary of each lenslet. The number of the lenslets required to represent a pixel is calculated as . Even though the scene created by the tomographic display contains the information matched with the RGB-D image in a one-to-one ratio, the optical pickup process allows the generation of the light field mapped in a one-to- ratio like the conventional InIm. As such, the proposed design can effectively perform the InIm projection with times higher resolution.
Fig. 3.
Experimental results of EI generation. (a) Captured EI at the screen plane. The white wheel image is sampled differently according to the pickup distance (Visualization 1). (b) Experimental MTF results when is . Binary gratings are captured using CCD camera without lenses mounted on.
Download Full
Size | PDF
We have measured the modulation transfer function (MTF) to analyze the resolution of the , as shown in Fig.
3
(b). A method of displaying binary gratings was utilized in the two cases of . Compared to the simulation result from the angular spectrum method, it fits quite well except for a small mismatch due to the lenslet aberration. Where the MTF is 17%, the prototype can generate the EI with a resolution of up to (28.6 megapixels). Even if we only use the relatively lower resolution LCD (0.22 megapixels) and DMD (0.58 megapixels), 36 times higher resolution is obtained. Consequently, our method can realize the projection-type InIm, which features ultrahigh resolution that previously could not be achieved with a display panel.
We constructed the MFPs for verifying the support of full-parallax and adequate focus cues. The results are captured at a distance of 1.5 m using a 50 mm focal length camera lens with . The duty cycle of the DMD projection for each depth is set to 0.1 [11]. Detailed image sequences are described in Supplement 1. As shown in Fig.
4
(a), the proposed method can support continuous parallax not only in the horizontal direction but also in axial direction within a viewing zone. Figure
4
(b) shows the experimental results of volumetric scenes. By changing the focal length of the camera lens, it was confirmed that the depth information of the volumetric scene is well reconstructed (See Visualization 2). Since the EI is supersampled and relayed directly, it represents an antialiased image in the focus plane.
Fig. 4.
Experimental 3D results. (a) Parallax according to the horizontal and axial distance. Five circles were constructed with a given RGB-D image. To emphasize the parallax, the images sliced horizontally are shown on the right. The results verify that the system can support full-parallax. (b) Volumetric scenes of Pieta and Market [18] captured with front and back focus (source image courtesy of “Pieta” www.cgtrader.com). The magnified images demonstrate the validity of 3D reconstruction (Visualization 2).
Download Full
Size | PDF
We utilize a Siemens star target to evaluate the contrast and imaging performance qualitatively, as shown in Fig.
5
(a). Since each spoke is located at a different depth, out-of-focused spokes are gradually blurred as getting away from a focused spoke marked with a red arrow. From these results, it is clear that the depth information from the MFPs is well transmitted via the EI. For the quantitative evaluation of the resolution, the experimental MTF curve for the reconstructed image is illustrated in Fig.
5
(b). The binary gratings in the same method of Fig.
3
(b) were utilized. Owing to the noise at the screen, we averaged the MTF near the center. The measured MTF is less than the wave simulation result, which is thought to be caused by aberrations in the projection lens and the MaLA. The results demonstrate that the proposed method can regenerate the light field on a large scale from high-resolution EI.
Fig. 5.
Experimental reconstruction results of the MFPs. (a) Qualitative results of Siemens star target for the green channel. Each spoke’s angle represents the reconstruction distance, and the radius from the center corresponds to the spatial frequency. The line trace along the arrow-line shows the contrast change. (b) Experimental MTF results for the reconstruction planes. Because of the inaccuracy by aliasing error near the MaLA, it was tested only for the distance where is .
Download Full
Size | PDF
In summary, a large amount of information demanded has been challenging to deal with in a glasses-free 3D display. In this Letter, we have proposed a new optical configuration that effectively brings spatial-angular information to the big screen. The method optically connects the multifocal display and the InIm display using a projection system. This is a perspective beyond the fundamental tradeoff relation rather than merely combining existing studies. With the novel design, our experimental results have demonstrated that optical pixel mapping can realize the EI close to 28.6 megapixels at 17% MTF and mitigate the information loss associated with the repetitive images in the InIm. Consequently, we have improved the resolution by 36 times, and it has been verified that a full-parallax volumetric scene can be implemented on a large scale through a projection. Our optical design could be widely integrated with existing light field displays, and high visual quality can be achieved based on previous projection-type InIm techniques. We hope that our perspective of optically manipulating spatial-angular information will inspire further developments for large-scale 3D displays.
Funding
Institute for Information and Communications Technology Promotion Planning and Evaluation Grant funded by the Korea Government (MSIT) (2017-0-00787).
Acknowledgment
This work is supported by the Institute for Information and Communications Technology Planning and Evaluation Grant funded by the Korea Government (MSIT) (development of vision assistant head-mounted display (HMD) and contents for the legally blind and low visions).
Disclosures
The authors declare no conflicts of interest.
Data Availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
Supplemental document
See Supplement 1 for supporting content.
REFERENCES
Archived URL: https://opg.optica.org/ol/fulltext.cfm?uri=ol-46-17-4212
�� CONTENT HASHES:
SHA-256: 5ff4cbbf8aaa283b5dfa4f32c878f737f67dedf7e55f0aceec410152e0c6b196
BLAKE2b: 473e0d88faf957c7d53cf80ee0fdf42cb8ccf4636da2da0d2066c45bb4eb9a42
MD5: 79487b62951c2b1c61bf3ee0bf21dbc7
�� TITLE HASHES:
SHA-256: b12587d680b53264ea2d132cd6a6a37d00538c878b6a8db361985189e3c7d66f
BLAKE2b: 4615b79000096cc8238d56072622793f282420f565248e9f657f00c1fa63e887
MD5: d712ca5004502b3ec6285780a4a82580
�� INTEGRITY HASHES:
SHA-256: ea0053d98cd385f5f3f3f9ff0865e91a55270261dcbad7d0688f53bbac594b45
BLAKE2b: cd0cb42e97df31b1b8f1fc84a991e57e81e045ca3d697e31d0cacb2c21bb2b9c
MD5: 465745cbaf60015c78de40b13c8ad95e
Archived with ArcHive - Client-side cryptographic archival system