Mosaic Collimated Display System

The following article was sent to me (Rick Lee) by John Amery of Boeing. He had read my article on collimated displays for PC simulators and wanted me to see some of the work he had done using fresnel lenses for professional simulators. I have posted the article here with his permission. This is fascinating stuff. Skip down to Figure 9 to get an idea of what it's all about.

FLIGHT SIMULATION VISUAL REQUIREMENTS and
A NEW DISPLAY SYSTEM

John G. Amery & Harry Streid

- John G. Amery: M/S S064 1481 Flight Simulation Technology
- Harry Streid: M/S S106 4685 Aerospace Training & Support Systems

The Boeing Company, PO Box 516, St. Louis, MO 63166-0516

ABSTRACT

This paper reviews the technical requirements for Out The Window (OTW) visual systems. Requirements for different modes of training and/or simulation will be stated. A new type of visual display will be described that provides improved, cost effective implementation and performance.

Keywords: simulation, out-the-window, flat panel display, COTS, continuous mosaic display, cost effective

1. INTRODUCTION

A new full-field-of-view visual system is under development, which promises to address many of the long-standing deficiencies in visual simulation for military aircrew training. Newly emerging requirements for training with aircraft flight HMDs and NVGs are addressed as well. Known as the Virtual Mosaic Display (VMD) this system is a high resolution, modular display system, fabricated with multiple commercial-off-the-shelf (COTS) Flat Panel Displays (FPD), combined as a continuous mosaic, virtual image, achieved with multiple fresnel lenses as the collimating optic. The display modules also include Personal Computers (PC) equipped with 3D Video Graphic Accelerator cards used as Image Generators (IG), interface to the FPD and communication to Host Computer. The fresnel lenses are arranged to tessellate a portion of the sphere as required to achieve the required field of view. Uncompromised full field of view representative of any modern fighter aircraft cockpit is achievable using this approach. Requirements for this system were quantified by examining the strengths and weaknesses of the wide-field-of-view visual systems, such as large dome front projected displays, flat screen rear-projected displays, and various other visual systems developed for tactical aircraft training over the last 20 years

2. VISUAL DISPLAY SYSTEM REQUIREMENTS for MODERN TACTICAL AIRCRAFT SIMULATORS

2.1 Determining Visual Requirements

The visual requirements to support aircrew training in a simulated flight environment have been studied for many years. These studies have been good and a wealth of data has been added to the science of human performance in the extremely demanding environment found in the modern fighter cockpit. Ideally, these studies would have resulted in the identification of a finite number of measurable physical quantities and minimum standards of acceptability for these quantities to perform the various types of training required. Operational users of these training systems could then specify their needs, which developers would then design to, the purchasers then buy and test to and users could then train to. There could be no question that simulator time would unequivocally lead to better safety in the actual aircraft and better performance in the tactical environment. Unfortunately, this is not yet the case. A great deal is now known about basic display parameters such as brightness, contrast, and resolution and their application to pilot performance in detecting and identifying a target, for example. Realization of visual systems which can provide adequate performance, however, especially over the very wide field of regard necessary with the bubble canopies on modern fighter and attack aircraft, has ultimately been limited by economics, if not technology.

2.2 Visual System Design is a Trade-off Exercise

When forced with economic and technological limitations, visual system designers are forced to make trade-offs. These trade-offs are usually made among the following variables¹ generally accepted as being most critical to visual simulation:

One of the variables most commonly traded off is field of view. Field of view may be traded off for resolution, update rate, brightness or collimation. The limitation in the number of pixels, which even the highest priced image generators can produce at a real-time rate, requires a trade-off in resolution vs. field of view. In addition, the physical limitation in constructing a large field of view, from traditional flight simulation display collimators, led to the use of large (20 to 40 ft diameter) dome style display systems, where the sheer size of the display screen limited the brightness of the scene projected upon it. The display system designer is left with a choice between a bright, clear image, but so limited in field of view as to be almost useless in a simulated tactical engagement, or wide field of view image which is so dim and fuzzy as to be almost equally useless. Collimation would usually be preferred, were its costs in dollars and in other visual simulation variables not extreme. In a collimated display, distant objects move along with normal pilot head movements and the simulated scene is always viewed in proper perspective. In addition, visual accommodation of the pilot's eyes, when focused on distant objects in a collimated display, is similar to that experienced in the real world. The transition time in visual accommodation, from head up to head down viewing of the aircraft instruments, is like the actual aircraft. One of the first full field of view visual systems for air combat military training was the Simulator for Air-to-Air Combat (SAAC) which used a mosaic of In-Line Infinity Optics "Pancake WindowTM" to achieve collimation over a wide field of view. The trade-off in brightness in the SAAC was immense due to the extremely low optical efficiency of the Pancake WindowTM. Typically though, training requirements for wide field of view displays have been met by real image systems.

2.2.1 Importance of Full Field of View

In the 1970s, a debate raged about the usefulness of force and motion cueing in tactical aircrew training. This debate was never fully settled. It was finally decided that the importance of a full field of view visual system was so overwhelming that motion cueing could be sacrificed if necessary to achieve full field of view. Air combat maneuvering and visual delivery of air-to-ground weapons are among the training tasks, which simply cannot be performed except with very large field of view display systems. Such large fields of view have mainly been achieved by projecting real images onto the inside of a spherical (dome) display screen.

2.2.2 Limitations of Domed Visual Systems

Many fixed-base flight simulators were equipped with domed visual system in the 1980s. The success of these systems was limited by another factor, which was often overlooked by the system designers but never by the pilots who had to try to train in these systems. The light reflected off of the inside surface of the dome display screen would simply bounce off the other side of the dome, become diffuse, and return to the projected image to reduce it's contrast. The greater the field of view in a dome, the worse the contrast. Increased quantities or higher lumen output projectors could improve the brightness but contrast remained limited. Switching from a diffusing screen to one with a more specular or retroreflective characteristic brought some relief, if the field of view requirements were not too large, but the demands for a lot of hand work to achieve adequate interior finish and coating of the large spherical areas, led to a lack of repeatability in the end result, not to mention high cost.

2.3 New Challenges for Visual System Designers

The end of the cold war brought about an end to much visual display experimentation as new aircraft starts were delayed and military training budgets were slashed. Technology continued to advance, however, and higher performance at lower cost became the norm as computational and graphics systems evolved. Display systems also advanced. As business and industry sought better ways to present data and graphics to large groups of viewers, smaller, lower cost, more reliable, high resolution projectors became available. The cost of military construction did not decline, however, and the large facilities required to house 40' domes, such as the one shown in Figure 1, were no longer built in the post-cold war era. A much smaller sized, full field of view, display system was needed.

2.3.1 Virtual Reality Brings No Help

For a time, it was felt that helmet mounted development, spurred by an anticipated consumer demand for "Virtual Reality" would result in low cost high performance head mounted visual systems which would be suitable for military aircrew training. The promise of helmet mounted displays has yet to be realized, however. This is partly due to the slow advances in the base technologies of miniature displays and optics and partly due to the decline of the over-hyped "Virtual Reality" market. To a large extent, it is also due to some very difficult human factors problems which limited user acceptance, at least within the military community. When configured to provide adequate field of view, resolution, and to allow normal visibility and interaction with aircraft controls and displays, helmet mounted displays were just too big and heavy and unnatural in fit
and form to be integrated into a tactical training environment.

2.3.2 Paradigm Shift

An alternate approach emerged from the Air Force Resource Laboratory. A Pancake WindowTM mosaic display known as the ASPT simulator (similar to the SAAC) was becoming difficult to maintain. Seeking a low cost, full color replacement for the dim, monochrome Pancake WindowTM led to experiments with rear projection screens and CRT projectors. A bright, clear real image was formed at approximately arm's length from the pilot's eye. One example of this type of display is shown in figure 2. Initial pilot evaluations are positive. A paradigm shift was about to occur. Collimation had been traded off so that adequate brightness, contrast and full color could be achieved in a full field of view display, which did not require new construction to accommodate. Visual experts everywhere were shocked. How could a non-collimated display ever be used to train pilots? Apparently, pilots thought it was more important just to have a display that actually provided the information they needed to execute their simulated tactical engagement than that it be presented exactly like the real world. There was still a problem with resolution, but when it was later found that high resolution target images could also be projected onto the close proximity display screens and the whole 360 degree system packaged to fit under an 8' ceiling, a very satisfactory visual system became available for visual air-to-air training. The full field of view visual system that motion cueing had been sacrificed for, many years before, was finally available.

2.4 To Infinity ....... Again!

The capability to train pilots in basic daytime fighter maneuvering currently exists. A full mission training capability does not. Such a capability would include full color at eye limiting resolution for ground targets to allow training in the most challenging visual tasks such as differentiating the scud launcher from the school bus. Full mission requirements would also include performing such tasks while wearing night vision goggles and/or a helmet mounted display and cueing system. Additional capabilities might include multi-spectral sensor augmentation of the target and displayed on the HUD or an HMD. To accomplish all of this, the visual system not only needs full field of view and high resolution, but it needs to become an integral part of the simulated weapon and sensor systems. Integrated detection and aiming cues must be provided to the pilot using off-boresite weaponry. To do this, the HMD imagery and Helmet Mounted Sight reticle imagery needs to maintain proper registration with the HUD and out-the window imagery even for the smallest head movements. Night Vision Goggles (NVG) have fast objective lenses which result in poor depth of field when actual NVG are used in the simulator with close proximity, real image displays. Problems in integration of HUD, HMD, and NVG into visual systems with physically close rear projection screens can all be solved, but not without incurring significant increase in cost and complexity. Clearly, a new type of display system is required for the 21st century, a display system that provides full collimation at eye-limiting resolution in full color over the full field of view of an advanced aircraft. Table 1 summarizes the preferred performance values for such a visual display system.

VISUAL DISPLAY SYSTEM VARIABLES	DESIGN GOALS	VMD CAPABILITY
Display Field of view	Full Field of View	Meets
Resolution	2 arc-minutes/line pair	Meets
Brightness	10 foot-lamberts	Exceeds
Contrast	10:1	Exceeds
Refresh & Update	60 Hz	Meets
Virtual (collimated) / Real Image	Collimated	Meets
Color	24 Bit RGB	Meets

3. A New Display System----Virtual Mosaic Display

3.0 Introduction

This section gives details of how the VMD System meets or exceeds the design goals. It also describes other desirable features and explains how it works. (is this enough)

3.1 Improved Performance

3.1.1 Resolution

Compromises in resolution for cost, size, mobility and feasibility have been made since visual simulation has begun. Different applications require different levels of resolution. The VMD also must be compromised for all of these same reasons. However, the VMD can provide better resolutions with available components today, at affordable costs. The VMD can be designed using different quantities of FPDs, having different numbers of pixels and with different sizes of displays. These variables of choice cause the whole display system to be different physical size, different resolution and of course different cost. The Feasibility VMD, as designed, has better than two arc minutes per pixel resolution. The VMD can be built to provide eye limited resolution.

3.1.2 Brightness

The brightness of the VMD is very good depending principally on the brightness of the FPDs. Typical Liquid Crystal Displays (LCD) available today provide peak white brightness of 200 cd/m2 . There is very little loss in the fresnel lenses, so most of this brightness is available to the viewer. This is a large improvement over many of the display systems in use today.

3.1.3 Contrast

Measured contrast on the Feasibility VMD is in the range of 50 to 100. It is principally the same as the FPD, in this case an LCD. This may be reduced somewhat in a full field of view display system. However, contrast of the VMD system is a significant improvement over many types of display available today.

3.1.4 Virtual Image

Real images from as close as 24 inches to as far as 20 feet are used effectively for Simulation and Training. However, it is more realistic and effective for training if the image is virtual, at or near infinity. The VMD provides a virtual image as shown in figure 3. The fresnel lens is a suitable distance away from the eye to allow freedom of movement, but closer to the imaging FPD than its focal length. These distances are designed to create a virtual image at or near infinity, of the real image on the FPD. The observed area on the FPD is less than the total area. The extra visual area on the FPD provides an image that overlaps adjacent images. Overlapped images are provided throughout the observation as shown in figure 4. The overlapped image allows body, head and eye movement. There is a trade off between resolution and allowable movement. The more movement allowed, the lower the resolution realized by the system. An effective compromise must be made.

3.1.5 Field of View

A VMD system can be configured into any field of view. Resolution is unaffected by the field of view, depending only on the resolution of the FPD in use. The major improvement of the VMD system is that it provides increased field of view at incremental costs. Full field of view up to 360O solid angle can be provided.

3.2 COTS Components

The imaging component used in the VMD is a Flat Panel Display. The most commonly available today, the LCD is used. These components are available from several manufacturers. Because they are being used for such things as laptop computers, the quantities manufactured are large and the prices are low. Today, LCDs incorporate several generations of improvements. The IG that drives each FPD is essentially a PC driving a high performance video card. The VMD System includes other COTS components, the power supply, interface card and ethernet communication.

3.3 Cost Effectiveness

3.3.1 Image Generator

Conventional IGs have improved and become more cost effective over the past two decades. However, their cost per pixel is still relatively high. A full field of view system with high resolution utilizing many conventional IGs becomes very costly. The VMD System utilizes a PC, coupled to fast 3D Video Graphics Accelerator Card to become an extremely cost effective IG. Full color (24 bit) and 60 Hz refresh rates can be provided by present day PCs with good video cards.

3.3.2 Imaging Display

Displays, both projected images and direct viewed are not improving as fast as IGs have. Multiple projectors have been a common solution to creation of a real or virtual image. Projector costs are also becoming lower, but the FPD remains a lower cost image source as well as providing a more compact system. LCDs presently available provide 24 bit color and 60 Hz update. The FPD contributes significantly to the cost effectiveness of the VMD system. Resolutions from VGA to 1600x1200 pixels are available. Progress will continue on low cost COTS FPDs and they will be implemented into cost effective displays of the future.

3.3.3 How Effective?

Cost effectiveness should be evaluated with comparable performance display systems. However, there may not be a similar display system in operation today to compare with the VMD. Let's look at the cost of the VMD. Although we have not built production models, we can consider the cost by looking at the components. Some of our Simulation and Training applications require a full hemispherical Out the Window (OTW) display. Of course, certain directions need not be viewed, e.g. down toward the cockpit floor. The Feasibility VMD built was part of a 162-panel system. We estimate that about 120 of those panels would be required for a typical single seat, fixed wing, military cockpit simulation. As we look at each module, containing a PC, power supplies, communications, 3D Video Accelerator card, interface and the FPD, the group of components would cost five to eight thousand dollars ($5K - $8K) in quantities of over one hundred. The support structure and fresnel lenses have some cost but it is considered small compared to the module. The cost for a full field of view (dome like) display system might be no more than $1Million, including the IGs. This system, patterned after the Feasibility VMD, would have about two (2) arc minutes per pixel resolution. Most of our existing full field of view display systems do not have this quality of resolution, nor the brightness and contrast of a VMD, but cost much more. Therefore, it's difficult to compare similar systems. However, it is easily envisioned that the VMD can be very cost effective for a large field of view high resolution display system.

3.4 Reliability

We are using COTS components that have good reliability, as properly implemented. COTS parts may have more reliability than the expensive, hardened, aged and documented MIL spec parts of the past. Not all COTS components are equal, so we look at what is available and select the components that will give effective reliability. A properly designed system built with COTS components will have good reliability.

3.5 Maintainability (Modularity)

Although the Feasibility VMD was not built with modules, the production concept utilizes cubical modules mounted on the rear of the support structure. Each module will contain the PC, communication to the host computer (e.g. ethernet), power supply, 3D Graphics Video Accelerator Card, an Interface card to the FPD (if necessary) and the Imaging Display itself (e.g. an LCD). The only connection required to each module is AC power and an ethernet connection. Modules will be identical with an identification switch. All modules will be interchangeable, allowing replacement maintenance in only a few minutes. Each location for a module will have a geometrical identification. Any replacement module can be switch selected to operate from that point. This switch selected geometry address also provides the PC a visual angular offset from a reference-viewing angle. As the viewpoint moves through the database, all modules move the same amount, change angle the same etc. each with a different offset.

3.6 How it works

3.6.1 Flat Panel Displays

COTS flat panels available today have borders to accommodate signal multiplexers and row and column drivers. However, this border makes it impossible to create a continuous direct view mosaic of displays from multiple flat panels. One of the key aspects of the new display system is the method that allows a continuous virtual image with separated flat panel displays. Figure 5 shows how this border problem is solved. The eye's cone of vision through each fresnel lens is confined to see only the display part of the FPD; borders are effectively hidden.

3.6.2 Tessellating the sphere²

Another problem in creating a continuous image as a spherical (or part of a sphere) view is that of tessellating the sphere. Tessellation (breaking up the surface into multiple flat panels) of the sphere has been done for decades as geodesic domes. The typical answer is multiple triangles. This works satisfactorily for domes, but the imaging tessellation has a more complex requirement. Given that the imaging source is a rectangle (4x3 or 5x4), a triangular view does not efficiently fit within the rectangular area. By this, we mean, a triangle (a triangular tessellation), has less than 50% coverage of the rectangle. There would also be a spatial problem for the rectangular FPDs. The solution for tessellating the sphere for the VMD uses mostly hexagons and a few pentagons. Both of these shapes utilize a larger percentage of the rectangular image, improving the efficiency.

Most geodesic dome tessellations begin with the icosahedron (twenty equalateral triangles forming a spheroid). This works if you leave the tessellation as triangles. However, if you want to combine the triangles into hexagons, many of the division ratios will not allow this. Another method of tessellating the sphere begins with the dodecahedron (twelve regular pentagons forming a spheroid). The next step divides each pentagon into five triangles with all of the common intersections defining the sphere. See fig. 6. The resulting triangles can be further divided into 4, 9, 16, 25, 36, 49, . . . etc. Finally, the divided triangles are combined to form 12 smaller pentagons at the center of the original 12 pentagons and the rest of the triangles are combined into hexagons. Ideally, all of the hexagons would be regular and the same size. This is impossible to accomplish³ with 162 panels. However, we have been able to make the three different hexagons approximately the same size (within a few percent) and approximately the same shape (slightly squashed) which fits within a rectangular image with good efficiency.

The Feasibility VMD tessellation began with a dodecahedron, dividing each pentagon into five triangles, then dividing each triangle into sixteen (16) equal triangles. The resulting 960 triangles combine into 12 pentagons and 150 hexagons to tessellate the full sphere. See fig. 7.

3.6.3 A Larger Display

Funding was limited, so the Feasibility Display was built having only a six panel window of about 60O conical view. A larger display of any size can be built by combining full display segments of 11 panels. Some additional panels fill spaces between segments. Parts of full segments could be used to fill the field of view as much as required. Figure 8 shows a rear view of the Feasibility VMD, with the electronics side of LCDs showing. Figure 9 shows a view from the right front. Five hexagonal fresnel lenses can be seen surrounding a pentagon fresnel lens. The viewpoint at the center is perpendicular to each of the fresnel lenses.

There are several choices to be made in designing a VMD. Requirements for size and resolution of the display system will dictate dimensions, quantity and resolution of the FPDs required. It is difficult to design a system this way. Available FPDs could be considered first. PCs and 3D Video Accelerator cards are available. Some latitude exists in total size depending on the fresnel lens selected. Although a COTS fresnel lens was used for the Feasibility Display, new designs should expect to design a lens, requiring a custom mold. The cost for this tooling is relatively small compared to the total cost of many systems. The dynamic changes occurring to FPDs, PCs, Video Cards etc., will require a VMD design to use parts available today and be ready to change the design tomorrow to the latest components.

4. Conclusion

The VMD system improves almost all of the problems of displays for Training and Simulation. It uses cost effective, COTS components that have sufficient reliability. It brings Virtual Imagery and deploy-ability for military applications. We expect to see the VMD concept used in the new millennium for commercial and military simulation and training, as well as industrial and entertainment applications. United States Patent Application Serial No. 09/197,026 was filed November 20, 1998 covering this system.

ACKNOWLEDGEMENTS

We wish to give full recognition to others that helped bring this new display system to its present state of development. John A. Van Hoogstraten recently retired with 40 years service to McDonnell Douglas, now The Boeing Company, for the optical design. Michael E. Stockton, (Boeing), for all mechanical design, support and tessellation of the sphere. Jody W. Spencer (Boeing) for PC & Software support. Our teamwork with these engineers, designed it, put it together and made it work.

REFERENCES

1. H. Snyder "Display Design Variables to Low-Level Flight Simulation", Visual Research for Flight Simulation, W. Richards, pp. 82-97 Appendix E, Technical Report AFHRL-TR-82-6, Brooks AFB, Texas, Air Force Human Resources Laboratory

2. Joseph C. Clinton "Geodesic Math", Dome Book II, Lloyd Kahn (Editor) (out of print)

3. Euler, Leonhard (1707 - 1783)

________________________________________________________________
Correspondence: E-mail & phone: (a) john.g.amery@boeing.com (314) 233-7554
(b) harry.streid@boeing.com (314) 234-6342