DataMesh Japan announces the successful implementation of an XR projection tour at the planned site of the “Torch Tower,” a structure reaching approximately 390 meters in height located in the “TOKYO TORCH” block, a project designed by Mitsubishi Jisho Design. The collaboration also involved NTT Communications, NTT QONOQ, and DataMesh.
The XR projection tour aimed at verifying the visualization effects of the planned building through the use of Digital Twin and XR technology, utilized DataMesh Director, which was sold by NTT Communications in Japan. The event, held three times with a total of 75 participants, primarily involved stakeholders from Mitsubishi Jisho Design. The tour utilized 3D design data (BIM) to project the future Torch Tower at the construction site.
Projected TOKYO TORCH Torch Tower with DataMesh Director and verified appearance. Note that this image is as of April 2023 and may undergo changes in the future.Projected TOKYO TORCH Torch Tower with DataMesh Director and verified appearance. Note that this image is as of April 2023 and may undergo changes in the future.
Background and Effect of the Initiative – Interviews with Stakeholders
In the R&D Promotion Department, we conduct information gathering, research, and investigations related to new technologies. During discussions with NTT Communications and QONOQ on utilizing XR technology in design tasks, we were introduced to DataMesh’s XR technology and decided to conduct a trial verification.
Traditional design work involves confirming design details using 2D perspectives or models. However, understanding spatial arrangements can be challenging for the general public. By projecting a 3D visualization of the building’s exterior into the actual space using MR technology, it becomes possible for people to see the planned building in real scale from various locations, facilitating easy confirmation of harmony with the surrounding environment.
Moreover, projecting the exterior of the Torch Tower, an ultra-high-rise building, using MR technology enhanced spatial awareness even more than conventional building sizes. Using BIM data for MR projection was anticipated to accelerate and streamline the design process, leading to increased efficiency.
Koju Matsuda (松田貢治) – TOKYO TORCH Design Room Manager, Mitsubishi Jisho Design (三菱地所設計/TOKYO TORCH設計室長):
As societal demands and the role of architects continue to evolve rapidly, there is increased involvement or leadership from clients. Hence, designers have an increased responsibility to convey and explain architecture to clients in a more understandable manner. In the case of the Torch Tower, we have been utilizing BIM to advance the design process.
BIM allows us to share visual recognition of space with clients, even if they cannot read drawings. While traditional landscape simulations involve creating CG or videos for clients to review on large monitors, XR provides a more realistic experience for both clients and designers to share the planned building.
Looking ahead, we anticipate that XR technology will enable sharing of information that was previously challenging to predict, such as the impact of high-rise buildings on wind conditions and changes in the perception of light during disasters. This communication enhancement through XR technology is expected to contribute to the evolution of both parties involved.
The effectiveness of XR lies in the ability to share, in real-time, how the planned building will look once constructed, along with the ambiance of the surroundings. The projection of the Torch Tower allowed us to confirm life-sized images while experiencing the surrounding buildings, cityscape, human and vehicular movement, sunlight, and wind. This made the image of the building more “visible” than previously imagined, allowing us to feel the completed structure more realistically.
During this verification, the data volume of the BIM design data (3D data) was substantial, given the unprecedented scale of the ultra-high-rise building and the project was conducted at the stage of completed design and pre-construction. To ensure smooth projection on an iPad, significant efforts were made to reduce the data volume. Challenges included the inability to include certain obstacles like temporary fences and street trees on the demolition site in the 3D data, resulting in some lack of realism, especially around the base of the building.
Looking forward, we hope that advancements in the efficient lightweighting of data at later design stages and the incorporation of 3D data of existing objects (captured by cameras) will further increase the feeling of realism, expanding the possibilities for utilization.
XR Projection Tour Participant Survey Overview
Survey Details: Objective: Verification of the effectiveness of visualizing planned buildings through XR utilization. XR Projection Dates: April 12, 19, and 21, 2023. Participants: Total 75 participants. Survey Respondents: 57 individuals (Response rate: 76%).
Note: The original Japanese version of this article was published here.
Interested in more details? Watch the video (Japanese) below for remarks from key stakeholders.
The expressway network of Japan has been constantly expanded since the 1960s and its total road length is currently over 10,000km. On the other hand, more than half of the expressway network has been used for over 30 years and aging of structures such as road surfaces, bridges, and tunnels has emerged as a social issue. The dwindling number of expressway maintenance and management engineers in line with the decline in the productive population is also one of the significant challenges.
Given the circumstances, Nexco-East Engineering operating about 4,000km of all the expressways in Japan developed a training app for ETC equipment and tunnel emergency facilities using MR and digital twin technologies, which is implemented in its training center “Technical Training Center (TTC)” located in Takasaki, Gunma. The app serves as an efficient education and training tool to equip every engineer with necessary knowledge and skills to develop quality engineers.
Initiated MR tool development toward improvement and standardization of trainees’ levels of understanding
Since about 10 years ago, the NEXCO EAST group has projected the aging of expressways and maintenance and management labor shortage and promoted the Smart Maintenance Highway (SMH) project to match the existing expressway maintenance and management technologies that the group has acquired with ICT technologies. Through the project, operational DX has been advanced with proactive application of digital capacities to the field operations, e.g. conversion of paper-based drawing preparation into tablet-based tasks and analysis and utilization of accumulated data with Microsoft PowerBI. Assigned with the project, Mr. Tetsuo Hideshima, the Managing Director and General Planning Manager of Nexco-East Engineering said with regard to the project policies,
“We at Nexco-East Engineering ensure that drivers can safely use expressways by providing inspections and troubleshooting all day, all year. For continuous maintenance and management of expressways, it is critical to conduct thorough inspections and diagnoses without overlooking any tiny changes in facilities and structures. To that end, streamlining and labor saving based on ICTs are imperative.”
The company also opened TTC in 2012 as a training facility to enhance performance of expressway inspection engineers. For practical training, TTC offers a training environment equipped with real power receiving and distribution equipment, ETC (automatic toll payment system) equipment, and component samples that are exactly the same as those used on expressways.
With regard to the roles of TTC, Mr. Masayuki Namiki, the General Manager of TTC said, “At TTC, over 1,000 of our engineers participate in training programs every year. They go through training to enhance skills based on the internal skill check system to become experts of expressway inspection and maintenance, called ‘highway doctors.'”
Meanwhile, highway structures and facilities have become more and more complex along with the technological advancement and, despite the hands-on learning with such training materials, levels of understanding of the mechanisms and internal structures varied among trainees. This hindered standardization of trainees’ technical levels, constituting an issue for TTC.
The issue was particularly conspicuous in education on ETC equipment. “ETC equipment is composed of multiple devices and how ETC equipment operates was explained with extensive materials such as paper-based radio communication and control signal flow charts, PowerPoint slides, and chalk talks of instructors and then tested with actual ETC devices,” said Mr. Yoshitaka Nakamura, the Manager, Facilities Construction Work Division, Facilities Construction Work Department of Nexco-East Engineering.
“However, it requires considerable time to gain understanding of associations among the training materials. Also, radio communication and control signal flows are impossible to see, so we had to rely on trainees’ imaginations. This resulted in gaps in understanding.”
In response, the company focused attention on the MR (mixed reality) and digital twin technologies. Then a project to develop an “MR for ETC equipment training” was launched out of an idea of combining a 3D model of ETC equipment as well as infrared sensors, radio communication, and data communication routes (which are invisible in reality) with the ETC practice lanes to foster better intuitive understanding.
Endeavor to develop training materials that are easy-to-use for every trainee with no example available as a model
It was DataMesh that contracted with Nexco-East Engineering for its extensive achievements and experiences in the field of digital twins and XR. Mr. Kensho Kashimada, the Director, Marketing and Sales of DataMesh Japan Co., Ltd. looked back, “At that time, there had been no similar attempts at least in Japan. The point was how we could build up MR materials that are easy-to-understand for every trainee while there was no correct answer about what the outcome should be like.”
The first step they took to that end was employment the agile development approach based on a quick PDCA cycles, rather than waterfall-model development that relies on detailed specifications from the beginning. “We carefully proceeded with the development through repeated hearing sessions with the project leading department and TTC instructors who would actually use the tool in practice,” said Mr. Kashimada. They also received the same courses that TTC actually provides to trainees to gain knowledge of ETC equipment mechanisms and applied it to the system.
“By receiving lectures with oral explanations and 2D materials, we witnessed how difficult it is to understand the ETC mechanisms firsthand. We could also appreciate the potential value of the tool we were trying to create. So I think it was a very valuable process,” said Mr. Kashimada.
Thus, after the development period of about six months, accurate 3D models of ETC devices were made and the MR for ETC equipment training using 3D animation superimposed on ETC equipment that is visible via a tablet device or Microsoft MR device “HoloLens 2” was completed.
“Normally, accuracy of SLAM (Simultaneous Localization and Mapping) in positioning of MR superimposition depends on hardware performance. However, for this MR for ETC equipment training, SLAM features of DataMesh’s proprietary digital twin platform (FactVerse) were used to make SLAM accuracy levels of multiple devices uniform,” explained Mr. Kashimada. Ingenuities are made to realize superimposition of massive 3D data without dependence on device performance by using a technology to load necessary portions of data only.
Mr. Kashimada said those features provides trainees immersive experiences in the MR world without stressful events such as misalignment or freezing of 3D animation.
Mr. Namiki said, “It was really a tremendous work to explain invisible things verbally. But now data and radio waves are visible and this increased the level of understanding dramatically.”
Once getting used to the operational feeling of HoloLens 2, trainees can adjust the position just by reading a QR code on the display without taking any complex procedure. It was reported that instructors could acquire basic operation methods only through a one-day lecture provided by DataMesh.
The MR for ETC equipment training, which is well recognized by field operators, too, has been further improved based on feedback from instructors. Currently, additional features such as repeated playback of individual work steps and animation speed adjustment for in-depth explanation make the tool even more effective.
MR for tunnel emergency facilities training making full use of HoloLens 2 characteristics
Out of its confidence in the success of MR for ETC equipment training, Nexco-East Engineering promptly commenced next training MR development, which is the “MR for tunnel emergency facilities training.”
“The internal structure of automatic water sprinkler system for tunnel emergencies cannot be seen from the outside, so we superimposed a CG simulation of the internal water flow and developed training materials for visualizing internals and operating status,” said Mr. Nakamura.
Mr. Namiki said, “In the past, we presented pipe samples from real equipment and illustrations to teach their structures and operations, but we couldn’t actually let water through them to demonstrate the mechanism. Also, explanations somewhat varied among instructors. As with ETC equipment, the gap in levels of understanding among trainees was recognized as an issue.”
As a measure to resolve the issue, DataMesh reproduced the equipment movements from an onset of vehicle fire to watering and visualized the internal structures based on its know-how acquired through development of the MR for ETC equipment training. Furthermore, a feature to adjust transparency of 3D animation of water flowing inside the pipe was added give trainees freedom to observe how water flows and stops inside. Implementation of realistic 3D effects of fire also give realistic experiences to enhance understanding.
Finally, quality of this MR for tunnel emergency facilities training is surprisingly high and precise even to experienced engineers including Mr. Hideshima. “From water flow and the valve opening mechanism to things going on behind pipes are all comprehensible at a glance. I sensed the poser of MR anew,” said Mr. Hideshima.
As the MR for tunnel emergency facilities training is equipped with the “pseudo reproduction feature” to learn points to check in the event of any malfunction and restoration methods, the MR system has a possibility to be applied to a future remote maintenance system.
Plus, linking tablets and other devices has enabled remote training, creating an environment where a greater number of trainees can take part.
Above all, the MR for tunnel emergency facilities training fully taps HoloLens 2 potential compared with the MR for ETC equipment training. In fact, for training using the MR for ETC equipment training with outdoor ETC equipment, tablets are mainly used as output devices because visibility of HoloLens 2 can be deteriorated depending on sunlight intensity. On the other hand, tunnel emergency facilities are indoor equipment and not subject to sunlight. This is where HoloLens 2 comes in.
Mr. Namiki said, “While a tablet needs to be held with both hands, HoloLens 2 is a head-mounted display, which sets both hands free. So trainees can touch devices while going through MR training. It also allows individual trainees to rewind, pause, and adjust the transparency level, giving an advantage in tailoring training sessions to individual levels of understanding.”
Toward even more effective ICTs through professional collaborations
After realizing the quality training tools by using the digital twin and MR technologies, Nexco-East Engineering has a plan to further promote use of ICTs to address the issues of facilities aging and labor shortage, which are expected to be further accelerated.
Exploring possibilities of ICT usages in various scenes, Mr. Nakamura said, “We have an ambition to expand our use of ICTs beyond TTC hands-on training to online training, OJT training, establishment of hands-on safety education system to improve field safety management skills, and remote maintenance, etc.”
With a view to establishment of a system for effective use of ICTs, Mr. Namiki struggling in the field of training said, “In recent years, equipment and facilities have become more and more sophisticated and suffered less failures. It may appear to be a good thing, but on the other side of the coin, most engineers in the future could be inexperienced in failure responses. We would like to establish a system to support engineers with the power of ICTs rather than relying on experienced engineers when the time comes.”
On the other hand, Mr. Hideshima is hopeful about professional partnerships. He said, “Although we have excellent expertise in expressway technologies, we admit we cannot fully catch up with ever-improving ICT seeds and technological innovations. In that regard, we are really hopeful about continuation of partnerships with ICT professionals such as DataMesh and Microsoft Japan.”
Based on ideal partnerships in which expressway professionals and ICT professionals work together to resolve issues and mutually give valuable input to technologies and knowledge, Nexco-East Engineering will continue maintaining and advancing safety on expressways, one of the essentials of our modern lives.
“From water flow and the valve opening mechanism to things going on behind pipes are all comprehensible at a glance. I sensed the poser of MR anew,”
— Mr. Tetsuo Hideshima, Managing Director and Chief of Planning Headquarters, Nexco-East Engineering
“While a tablet needs to be held with both hands, HoloLens 2 is a head-mounted display, which sets both hands free. So trainees can touch devices while going through MR training. It also allows individual trainees to rewind, pause, and adjust the transparency level, giving an advantage in tailoring training sessions to individual levels of understanding,”
— Mr. Masayuki Namiki, General Manager, Technical Training Center Manager, Nexco-East Engineering
“We have an ambition to expand our use of ICTs beyond TTC hands-on training to online training, OJT training, establishment of hands-on safety education system to improve field safety management skills, and remote maintenance, etc.”
— Mr. Yoshitaka Nakamura, Manager, Facilities Construction Work Division, Facilities, Construction Work Department, Nexco-East Engineering
“At that time, there had been no similar attempts at least in Japan. The point was how we could build up MR materials that are easy-to-understand for every trainee while there was no correct answer about what the outcome should be like.”
— Mr. Kensho Kashimada, Director, Marketing and Sales, DataMesh Japan Co., Ltd.
This customer story was originally published by Microsoft here.
Obayashi Corporation (株式会社大林組) has successfully introduced DataMesh Director, a DataMesh digital twin platform service, to empower frontline personnel with BIM data combined with mixed reality technology to improve construction processes’ efficiency on its construction sites. Behind the scene, this is a collaboration result between DataMesh Corporation and TIS Corporation. Obayashi Corporation is one of the top five general construction companies in Japan, ranking first in total assets and revenue.
Figure 1 // Musashi Kosugi Station construction site – Confirming the construction sequence of the reinforced inclined beam by MR technology
Due to the population aging and low fertility rate, the number of employees in Japan’s construction industry is decreasing. A large number of employees will likely leave the industry in the future. The construction industry’s unique characteristics lead to longer working hours than other sectors, and employees often work on weekends. To ensure the continued participation of young workers, the most critical issue in the construction industry is how to improve work efficiency and solve the problem of long working hours.
For this reason, Japan’s Ministry of Land, Infrastructure, Transport, and Tourism has proposed the i-Construction Policy, which calls for flexible use of ICT technology and BIM/CIM resources, and plans to increase labor productivity by 20% by 2025.
The project led by TIS and DataMesh provides DataMesh Director, a platform for rapidly building 3D construction process, successfully landed the practical application of BIM/CIM model combined with MR technology to improve work efficiency.
Construction Efficiency Improvement
With the gradual increase of city operation management requirements, the city needs to carry out real-time construction since the operation process. The construction contractors’ time slot will be allocated within a few hours after the subway stops running at night. The construction time is extremely tight. If problems arise, existing jobs need to be reworked in time. Otherwise, huge losses may occur.
In previous construction experience, a great deal of time was spent between the commissioner and the contractor’s staff to strictly confirm the construction details to ensure the correct construction process.
In this project, the staff of the Obayashi Corporation imported BIM data by using DataMesh Director. He quickly created the construction process construction in 3D motion. He then used the Mixed Reality hardware devices such as Microsoft HoloLens 2 and iPad to accurately map the construction process to the complex construction site through DataMesh One with spatial positioning. The overall construction process is confirmed.
Jessica Miao, Mixed Reality Asia Pacific Lead at Microsoft, said, “Microsoft highly appreciates this collaboration between the Obayashi group, TIS, and DataMesh in construction projects. In the face of the social problem of decreasing workforce, DataMesh Director realized the combination of MR products and BIM data based on Microsoft HoloLens 2 Mixed Reality headset for hands-on guidance of frontline personnel. This will also become a real-time construction and data sharing method in the field of smart cities. We hope that with this pilot, we can form a typical application effect of smart cities globally and further improve the construction of smart cities and the productivity of the construction industry.“
This deployment is divided into two projects as follows.
Railway Bridge Construction Project – Yokohama Loop South Line between Totsuka and Ofuna on the Tokaido Main Line
In the railway bridge construction project, the nine lines operating in real-time around the construction needed to be temporarily blocked after construction. The construction crew had to complete the bridge’s mainframe within 100 minutes to ensure that it would not cause massive traffic disruption.
Obayashi Corporation staff imported BIM data to customize the construction process through the digital twin platform DataMesh Director. He quickly designed the process through zero-code tools, with HoloLens 2/iPad/iPhone to present the engineering materials and construction steps combined with spatial positioning. The engineers onsite can now quickly discover the problems and risks in construction and significantly improve the construction project’s efficiency.
Fig. 2// On-site MR effect of bridge construction project
Station Hall Expansion Project – Musashi Kosugi Station on the Yokosuka Line
In the station concourse expansion project, the inclined beams used for reinforcement during construction were exposed in the temporary passageways for passengers, and the impact on moving passengers had to be considered when the project was planned.
The operations staff of the Ohayashi Corporation imported BIM data through DataMesh Director, created the expansion process. Onsite construction workers brought HoloLens 2/iPad to quickly understand the construction process and scale of the project through Mixed Reality, matched the site environment to observe possible construction problems, and strictly controlled the risks that might be caused to the passing passengers during the actual construction.
Fig. 3// Reference schematic diagram of reinforced inclined beam construction
This project was highly recognized by the Obayashi Corporation as follows.
“The inclined beam ground reinforcement was to be carried out in a very narrow temporary passage for passengers, and the planning of this construction by MR technology was effective in allowing the commissioning party and the staff involved in the construction to understand the construction process and other aspects better.” Keita Koshida, director of the Musashi Kosugi Station Public Works Office, said. “The steel assembly process is presented in MR, allowing even unskilled workers to understand the sequence of real-time construction well, which can solve the problem of lack of human resources to some extent. the application of MR provides a new means of visualization for the construction site, which is very effective in improving site safety. “
“By visually presenting the process of assembling the main girders of the bridge, it facilitates understanding the construction engineering plan and the in-depth planning of future construction plans, thus better illustrating the frontline site situation to the commissioning party. MR technology will play an important role in expressing the project planning, confirming the situation of construction impact on existing and temporary buildings, construction conditions and changes in the surrounding environment after construction.” Yoichi Nishikori, the chief engineer of Yokohama Kanjo Minami Bridge Works Office, said, “When the technology is further developed, MR will also be able to be used for more detailed work, such as site verification and construction release. Making effective use of ICT technology and BIM/CIM can make the business of actual construction more efficient.“
For his part, Takafumi Yamanaka, director of the apex technology planning department at the Oyashi Group Production Technology Headquarters, said, “Before that, we had no available tool for visualizing MR construction processes in construction, and now DataMesh Director enables this function. It is a milestone that DataMesh Director can be used by people who are not professionals and who can create content with it from scratch. There are many situations where DataMesh Director can be used on construction sites, and we will continue to communicate with TIS and DataMesh with each other to make DataMesh Director a tool that can be used by everyone on various construction sites.“
DataMesh Director
DataMesh Director is a self-developed digital twin content creation and collaboration platform based on the concept of “XR content tools for everyone”.
During modern city construction, the actual construction process varies in different construction plans. The number of construction workers is highly mobile and irregular, so there is a demand for a communication method to better integrate with the construction plan.
Combine BIM/CIM resources with the construction site to effectively utilize the virtual reality In traditional construction sites, detailed process guidance or location confirmation is usually only available in plan drawings. By directly importing existing BIM/CIM resources to DataMesh Director, the construction process can be precisely combined with the real space through mixed reality devices supporting spatial positioning and output as a three-dimensional process in front of frontline personnel. It can effectively ensure that site personnel have a more accurate understanding of the complex site environment and project progress and prevent construction risks. The combination of virtual and real presentation also allows novices to understand the business process quickly and effectively improve construction efficiency.
Zero technical thresholds; everyone can easily create and present 3D MR content By importing BIM/CIM resources, business personnel can use DataMesh Director to quickly edit the construction process into 3D scripts that can be mapped in real space through the operation similar to editing PPT, with low learning cost. It can be published to multiple ends with one click for remote multi-person collaboration. The content production method is simple and can be modified and reused at any time.
Support HoloLens 2, iPhone/iPad, generic XR glasses and mobile devices DataMesh Director supports various content presentation devices, freeing the hands of frontline personnel. From HoloLens 2 and other mixed reality headsets to mobile devices, DataMesh Director supports real-time instructions sharing and cross-device collaboration, efficiently improving construction sites’ communication.
Strategic Cooperation Parties
DataMesh (https://datamesh.com) is a high-tech startup that uses MR+AI to create a digital twin platform. As a Microsoft Gold Partner, DataMesh is serving customers across China, Japan, Southeast Asia, and North America. DataMesh is committed to unlocking the value of enterprise data, building bridges and platforms that connect the virtual and real worlds, and improving frontline employees’ efficiency in industries such as manufacturing, energy, construction, and professional services.
TIS (http://www.tis.co.jp) provides service-oriented IT solutions such as data centers and clouds, in addition to system integration and development support. We have also formed a global support system centered on the China-ASEAN region, contributing to our customers’ business growth as a partner of more than 3,000 companies in a wide range of industries, including finance, manufacturing, distribution/service, public, and communications.
