In today’s fast-evolving manufacturing landscape, traditional approaches struggle to keep up with the growing complexity of production workflows and the demand for continuous optimization. Data silos, complex integrations, and slow response times hinder not only operational efficiency and maintenance but also limit long-term digitalization strategies.
The rise of digital twin technologies has paved the way for new solutions. DataMesh FactVerse, an industrial metaverse platform built on advanced simulation digital twin technology, enables manufacturers to rapidly create digital twins, scenarios, and workflows. With its powerful data fusion engine (FactVerse DFS) and AI-powered content generation module (FactVerse AI, or FAI), FactVerse streamlines content and application creation, accelerates deployment, and seamlessly integrates real and virtual environments, unlocking unparalleled productivity and value.
The NIO Success Story
In collaboration with DataMesh, NIO deployed a Production Management Center (PMC) digital twin platform for its vehicle assembly line. Powered by FactVerse, this platform transforms production management by enabling comprehensive simulation and real-time monitoring of the entire production process within a highly detailed, 1:1 digital twin of the factory.
PMC Management Interface of the Digital Twin Platform
Key Highlights
Effortless Twin Creation: Engineers can quickly create digital twins of production lines and equipment, integrating 3D models and workflows for precise simulations and real-time monitoring within a virtual environment.
Production Layout Simulation: Teams can adjust production line layouts, simulate operations with high accuracy, and optimize AGV routes and equipment logic—all without requiring any programming skills.
Data Integration and Real-Time Monitoring: The FactVerse DFS module seamlessly integrates data streams from diverse sources, including AGV locations, RFID systems, and the Production Execution System (PES). This creates a seamless virtual-to-physical connection, enhancing monitoring and enabling rapid resolution of production bottlenecks.
This innovative platform has demonstrated the unique value of digital twins in NIO’s vehicle assembly operations. By enhancing transparency, streamlining monitoring, and fostering smooth collaboration, the platform has significantly reduced downtime and minimized production disruptions. This successful implementation has established a new standard for digital innovation in manufacturing.
The project lead from NIOemphasized: “This digital twin project has not only delivered tangible results in our current production processes but also laid a solid foundation for future advancements. The user-friendly nature of DataMesh’s digital twin technology empowers our team to easily simulate, monitor, and optimize workflows with unprecedented efficiency. We are excited to continue exploring its long-term potential and leveraging it to drive sustainable growth at NIO.”
Accelerating Digital Transformation with FactVerse
FactVerse is an industrial metaverse platform focused on using simulation digital twins (SDTs) to create real-time, intelligent, and interconnected business scenarios. Ideal for manufacturing, smart facility management, and sustainable construction development, FactVerse addresses critical challenges such as spatial data processing and real-time 3D collaboration, driving digital transformation across industries.
Key Advantages of FactVerse
Advanced Data Model Management: Unifies multi-source data into a single, consistent model to provide real-time insights and comprehensive views.
Efficient Digital Twin Creation: Features an intuitive drag-and-drop interface with industry-specific DLCs (downloadable content), such as digital twin factory, warehousing, and logistics, for rapid scene creation.
Interactive OOTB Applications: Offers out-of-the-box fronline applications like Director, Checklist, Inspector, and Simulator to optimize training, troubleshooting, experience, monitoring, control, and simulation (TEMS).
Data Fusion Services: FactVerse DFS enables seamless integration with critical business systems, ensuring efficient data standardization.
AI-Powered Assistance: FactVerse AI provides intelligent search and expert guidance powered by large language models, bridging the gap between human and machine interactions.
Compliance and Standardization: Supports green and industry standards, including energy calculation tools, to help organizations meet sustainability goals and reduce their carbon footprint.
FactVerse continues to empower global leaders like NIO to achieve their sustainability and operational goals, driving efficient, sustainable, and low-carbon development for a better future.
Watch the video to explore how FactVerse supports NIO in building their digital factory and advancing digital transformation with digital twins.
In today’s highly competitive environment, digital transformation is crucial for manufacturers looking to enhance competitiveness and achieve long-term growth. As a global leader in electronics manufacturing, Foxconn is embracing cutting-edge technologies such as digital twins, artificial intelligence (AI), and extended reality (XR) to improve productivity, product quality, and operational efficiency while reducing costs and resource consumption.
To further strengthen its competitive edge, Foxconn is heavily investing in technology to digitize its past data and experiences. By creating digital solutions tailored to its factory operations, Foxconn is setting new industry benchmarks. The company selected DataMesh FactVerse—a state-of-the-art digital twin platform—and Apple Vision Pro, a next-generation XR device, to enable its frontline employees to effortlessly integrate these technologies into their daily work, enhancing overall efficiency.
Tailored Digital Solutions for Factory Operations
The FactVerse platform allows enterprises to quickly build digital twins of their factories, offering essential tools for real-time monitoring, simulation, intelligent training & maintenance, etc.
By integrating FactVerse into its business processes, Foxconn is transforming its factory management system, boosting operational efficiency across the board. FactVerse-based applications support various devices, including Apple Vision Pro, iPads, and PCs, ensuring seamless access to information, intelligent recommendations, and real-time guidance for employees. This results in faster execution, fewer mistakes, and improved production outcomes.
In the first phase of implementation, Foxconn focused on two key areas: employee training and equipment maintenance. With the help of DataMesh Director (a PPT-like 3D MR instruction creator for training and guidance) and DataMesh Checklist (a step-by-step validation tool), frontline workers can create MR training content and inspection plans easily. By utilizing XR devices like the Apple Vision Pro, employees can complete training and inspection tasks in a simpler, more intuitive way—greatly enhancing work efficiency.
Enhancing Employee Training with DataMesh Director and Apple Vision Pro
Challenges
High Training Demands: Foxconn’s factories involve large-scale production lines with complex processes and numerous workstations. With a large influx of new employees, training resources are often stretched thin.
Frequent Training Needs: Labor shortages and high turnover rates necessitate frequent training sessions, placing additional strain on trainers and resources.
Steep Learning Curve: With a variety of workstations and complex machine operations, new employees face a steep learning curve. Trainers with expertise in a single skill cannot comprehensively guide all new employees, resulting in longer training periods.
Ineffective Learning Outcomes: Limited access to physical machines and a lack of interactivity in traditional remote training methods often result in poor learning outcomes.
Solution
Foxconn trainers can use DataMesh Director to quickly import 3D models and create MR training materials without any programming. This allows new employees to access training content anytime and anywhere, using computers, smartphones, tablets, or XR devices. They can engage with virtual machines in an interactive environment, significantly speeding up the learning process.
Cross-Device Collaboration
Apple Vision Pro further enhances MR training by providing high-definition visuals that closely replicate real machines and operational steps. Trainees can interact with these virtual machines, gaining a deeper understanding of training content. This immersive experience improves learning efficiency and retention.
Gesture-Based Interaction
The 3D operation manuals created through the platform can be reused in various training scenarios, such as job training, equipment maintenance and inspection, and skill development, making the learning process more flexible and effective.
Looking ahead, the FactVerse platform will provide Foxconn with an employee assessment system, which will gather data from training processes and performance results and combines them with the comprehensive goals of production schedules, job designs, and process designs to optimize training plans and further enhance employee productivity.
Key Benefits
Improved Training Efficiency: Job training efficiency increased by 30%.
Resource Savings: The no-code approach shortens the development cycle and cuts down costs for training materials. Employing 1:1 virtual machines in place of the real ones for training, reducing equipment consumption costs.
Reduced Learning Time: 3D and MR training content offer clearer, more interactive learning experiences, speeding up the learning process.
Digital Asset Accumulation: The knowledge and experience converted into digital assets, creating a valuable resource for future training programs.
Improving Equipment Inspection with DataMesh Checklist and Apple iPad
Challenges
Complexity of Inspections: Traditional SOP (Standard Operating Procedures) documents can be difficult to follow, making inspections time-consuming and prone to errors.
Challenges with Remote Guidance: In remote environments, skill transfer is limited, and workers may not get the timely support they need, resulting in lower inspection quality.
Solution
Using DataMesh Checklist, Foxconn employees can easily create and manage inspection plans, linking them with 3D operation manuals. During inspections, employees can use iPads to view real-time guidance overlaid on the equipment, following step-by-step instructions to ensure accuracy and prevent errors. With the help of AI, workers can quickly access maintenance history and receive real-time operational guidance to perform repairs more efficiently.
Foxconn also plans to integrate the FactVerse platform with its production system and fault-warning system, enabling a data-driven approach to decision-making and further improving operational efficiency.
Key Benefits
Improved Inspection Accuracy: Equipment spot-checking failures have been reduced by 4%.
Cost Savings in App Development: Employees can use out-of-the-box tools to create inspection tasks, reducing the time and cost of developing new applications.
Alignment with Production Schedules: Coordinating inspection plans with production schedules ensures equipment reliability during peak times, preventing downtime and improving overall production efficiency.
Data-Driven Decision Making: The platform’s traceability enables better data analysis, helping to optimize production and equipment management.
Empowering Frontline Workers and Enhancing Workforce Engagement
Foxconn’s collaboration with DataMesh and Apple is transforming the work experience for its frontline employees, particularly Generation Z workers who are eager to engage with new technologies and learn new skills to improve work efficiency and income. This collaboration also provides significant advantages for Foxconn, such as reduced recruitment and training costs, lower employee turnover, and a more skilled workforce.
As Foxconn continues to work with DataMesh and Apple, it will leverage digital twins, AI, and spatial computing to further empower its employees, drive innovation, and improve production efficiency—ultimately achieving sustainable growth.
In May 2023, JTC successfully introduced DataMesh Director, a digital twin platform by DataMesh, to empower frontline personnel through the integration of Building Information Modelling (BIM) data with mixed reality (MR) technology on construction sites. This project aims to enhance efficiency by improving the visualisation and understanding of complex processes.
Discover how DataMesh Director was used to elevate construction sequences in this video
Project Overview
The collaboration between DataMesh and JTC led to the implementation of DataMesh Director, enabling the rapid creation of 3D construction processes. The platform uses digital twin and MR technology to turn BIM/CIM models into practical, real-time applications on construction sites, allowing workers to better visualise their tasks, understand project requirements, and improve overall workflow.
Figure 1: Confirming the construction sequence of the reinforced inclined beam using MR technology at the Pan Island Expressway construction site
Empowering Frontline Workers with Mixed Reality
A key benefit of using MR technology on construction sites is its ability to help frontline workers visualise tasks in both virtual and physical spaces. This is especially important in Singapore, where many construction workers come from countries like Bangladesh and face language barriers. MR technology empowers these workers to:
Visualise Complex Processes: Workers can gain a clearer understanding of the construction sequence and their specific roles by viewing 3D models and simulations directly at the site.
Improve Communication: MR technology overcomes language barriers, enabling workers to follow instructions effectively and understand project requirements from management.
Enhance Safety: Safety guidelines are presented in real-time, ensuring workers stay in safe zones and avoid risks, such as the lifting of beams.
Through the integration of DataMesh Director and MR technology, this project has set a new standard for improving productivity, communication, and safety on construction sites.
Figure 2: DataMesh Director construction sequence
Key Benefits of Using DataMesh Director for Construction Sequences
Enhanced Visualisation: With 3D visualisations of construction sequences, workers can follow complex instructions more easily, improving task execution accuracy and reducing errors.
Improved Communication: The platform effectively bridges language gaps, enabling workers to understand instructions and safety protocols, regardless of their native language.
Increased Efficiency: With a clear visual representation of the construction sequence, workers can perform tasks more efficiently, reducing the time spent on interpreting complex plans and avoiding rework.
Enhanced Safety: Real-time safety guidelines ensure that workers stay in safe zones and avoid dangerous situations, minimising the risk of accidents.
Real-Time Problem Solving: The integration of IoT data allows for real-time monitoring and adjustments. If any issues arise during the construction sequence, they can be identified and addressed immediately, reducing downtime and preventing costly rework.
By providing clear, interactive, and real-time visualisation of construction sequences, DataMesh Director has proven effective in enhancing the construction process on numerous customer projects across the Asia Pacific region.
Figure 3: DataMesh Director MR overview of hoisting construction sequenceFigure 4: DataMesh Director MR overview of hoisting construction sequenceFigure 5: DataMesh Director MR overview of construction sequenceFigure 6: JTC’s CTPPOB construction at 92-percent progress
DataMesh Director Capabilities: Facilitating the Digitalisation of Construction Workflows
DataMesh Director enables users to create 3D digital twin content easily through a user-friendly drag-and-drop interface—similar to creating a PowerPoint presentation. The platform allows users to quickly leverage existing 3D assets, like CAD/BIM files, to build digital twins, improving the speed and efficiency of construction workflows. Key features include:
Ease of Creating Digital Twins: Quickly develop 3D digital twin content using existing assets, streamlining the content creation process.
IoT Data Binding: Seamlessly integrates with IoT devices for real-time data visualisation.
Remote Collaboration: Supports collaborative work environments, enabling multiple users to work together regardless of location.
Cross-Platform Content Publishing: One-click publishing makes it easy to share content across various platforms.
AI Support: Integrates FactVerse AI (FAI), a generative AI tool, for the effortless creation of multi-language scenarios based on enterprise digital twin data.
Mixed Reality Ready: Deploys to AR/MR-capable devices like iPhones, iPads, Apple Vision Pro, Meta Quest 3, and compatible Android devices with a single click.
This powerful platform enables the creation of job SOPs, support guides, 3D product manuals, and real-time equipment monitoring, significantly enhancing the efficiency and experience of frontline workers. Beyond construction, DataMesh Director is also valuable for aftersales support, marketing, exhibitions, training, and more.
DataMesh Japan announces the successful implementation of an XR projection tour at the planned site of the “Torch Tower,” a structure reaching approximately 390 meters in height located in the “TOKYO TORCH” block, a project designed by Mitsubishi Jisho Design. The collaboration also involved NTT Communications, NTT QONOQ, and DataMesh.
The XR projection tour aimed at verifying the visualization effects of the planned building through the use of Digital Twin and XR technology, utilized DataMesh Director, which was sold by NTT Communications in Japan. The event, held three times with a total of 75 participants, primarily involved stakeholders from Mitsubishi Jisho Design. The tour utilized 3D design data (BIM) to project the future Torch Tower at the construction site.
Projected TOKYO TORCH Torch Tower with DataMesh Director and verified appearance. Note that this image is as of April 2023 and may undergo changes in the future.Projected TOKYO TORCH Torch Tower with DataMesh Director and verified appearance. Note that this image is as of April 2023 and may undergo changes in the future.
Background and Effect of the Initiative – Interviews with Stakeholders
In the R&D Promotion Department, we conduct information gathering, research, and investigations related to new technologies. During discussions with NTT Communications and QONOQ on utilizing XR technology in design tasks, we were introduced to DataMesh’s XR technology and decided to conduct a trial verification.
Traditional design work involves confirming design details using 2D perspectives or models. However, understanding spatial arrangements can be challenging for the general public. By projecting a 3D visualization of the building’s exterior into the actual space using MR technology, it becomes possible for people to see the planned building in real scale from various locations, facilitating easy confirmation of harmony with the surrounding environment.
Moreover, projecting the exterior of the Torch Tower, an ultra-high-rise building, using MR technology enhanced spatial awareness even more than conventional building sizes. Using BIM data for MR projection was anticipated to accelerate and streamline the design process, leading to increased efficiency.
Koju Matsuda (松田貢治) – TOKYO TORCH Design Room Manager, Mitsubishi Jisho Design (三菱地所設計/TOKYO TORCH設計室長):
As societal demands and the role of architects continue to evolve rapidly, there is increased involvement or leadership from clients. Hence, designers have an increased responsibility to convey and explain architecture to clients in a more understandable manner. In the case of the Torch Tower, we have been utilizing BIM to advance the design process.
BIM allows us to share visual recognition of space with clients, even if they cannot read drawings. While traditional landscape simulations involve creating CG or videos for clients to review on large monitors, XR provides a more realistic experience for both clients and designers to share the planned building.
Looking ahead, we anticipate that XR technology will enable sharing of information that was previously challenging to predict, such as the impact of high-rise buildings on wind conditions and changes in the perception of light during disasters. This communication enhancement through XR technology is expected to contribute to the evolution of both parties involved.
The effectiveness of XR lies in the ability to share, in real-time, how the planned building will look once constructed, along with the ambiance of the surroundings. The projection of the Torch Tower allowed us to confirm life-sized images while experiencing the surrounding buildings, cityscape, human and vehicular movement, sunlight, and wind. This made the image of the building more “visible” than previously imagined, allowing us to feel the completed structure more realistically.
During this verification, the data volume of the BIM design data (3D data) was substantial, given the unprecedented scale of the ultra-high-rise building and the project was conducted at the stage of completed design and pre-construction. To ensure smooth projection on an iPad, significant efforts were made to reduce the data volume. Challenges included the inability to include certain obstacles like temporary fences and street trees on the demolition site in the 3D data, resulting in some lack of realism, especially around the base of the building.
Looking forward, we hope that advancements in the efficient lightweighting of data at later design stages and the incorporation of 3D data of existing objects (captured by cameras) will further increase the feeling of realism, expanding the possibilities for utilization.
XR Projection Tour Participant Survey Overview
Survey Details: Objective: Verification of the effectiveness of visualizing planned buildings through XR utilization. XR Projection Dates: April 12, 19, and 21, 2023. Participants: Total 75 participants. Survey Respondents: 57 individuals (Response rate: 76%).
Note: The original Japanese version of this article was published here.
Interested in more details? Watch the video (Japanese) below for remarks from key stakeholders.
Today, two billion frontline workers globally serve as the backbone of various industries, yet they face challenges such as insufficient technical training, collaboration difficulties, and health and safety risks. In developing countries, these workers have even more limited access to modern technology, necessitating innovative solutions for enhanced efficiency and safety.
DataMesh’s mission is to support frontline workers in the current era where technology and human creativity converge, utilizing digital twins and XR technology. In line with this mission, DataMesh introduced DataMesh Director, a 3D guidance tool based on digital twins and XR technology. It enables cost-effective and rapid training on complex equipment using smartphones or MR devices and allows frontline workers to customize operation tutorials without programming or 3D modeling knowledge.
In partnership with 100+ Accelerator and DataMesh, Swire Coca-Cola initiated a digital transformation project to enhance maintenance processes within its bottling facility. By implementing DataMesh’s digital twin and mixed reality technology, local workers received immersive training, enabling them to perform complex maintenance tasks with minimal downtime. This successful collaboration not only reduces operational costs but also empowers the workforce, driving productivity and job opportunities in the region.
In process manufacturing, especially in packaging lines within the bottling facility, training and maintenance are critical yet complex. The intricate disassembly and assembly of aseptic filling valves, traditionally time-consuming to train, are streamlined by DataMesh Director. It creates 3D instructions for local workers on aseptic filling valves, combined with augmented reality (AR) technology, allowing them to intuitively grasp operational procedures through smartphones, tablets, or MR headsets. With DataMesh’s assistance, workers receive real-time feedback, significantly reducing training time, and can now assist senior engineers more efficiently during actual work.
Innovative strategies extend beyond training to daily operations and maintenance. The new training model is proven to reduce expected maintenance downtime by one-third, enabling the production line to produce millions more beverage cans annually as expected.
Inclusive growth, a core element of ESG, is demonstrated in this collaborative project. DataMesh’s solution ensures widespread technology adoption and equalized training, allowing previously marginalized frontline workers to enjoy modern training and operational experiences. These equal opportunities, driven by technology like those from DataMesh, not only promote capability building and employee professional growth but also bring fast tangible benefits to businesses. This true inclusive growth, consistent with the philosophy of ESG, has proven to be a perfect combination of technology and ESG goals, setting a role model for the industry.
DataMesh showcased the innovative achievements in process manufacturing of this collaboration at the 100+ Accelerator 2023 Demo Day by AB InBev, presenting to renowned industry leaders like AB InBev, Coca-Cola, Colgate-Palmolive, Unilever, and more.
Nikita Huang, Chief Operating Officer of DataMesh, expressed gratitude for collaboration with industry leaders and the strong support from AB InBev’s 100+ Accelerator, looking ahead into the future, stating, “This is just the beginning. DataMesh, equipped with Digital Twin and XR technologies, remains committed to offering more convenience and support to frontline workers in process manufacturing, pushing the industry forward.”
The expressway network of Japan has been constantly expanded since the 1960s and its total road length is currently over 10,000km. On the other hand, more than half of the expressway network has been used for over 30 years and aging of structures such as road surfaces, bridges, and tunnels has emerged as a social issue. The dwindling number of expressway maintenance and management engineers in line with the decline in the productive population is also one of the significant challenges.
Given the circumstances, Nexco-East Engineering operating about 4,000km of all the expressways in Japan developed a training app for ETC equipment and tunnel emergency facilities using MR and digital twin technologies, which is implemented in its training center “Technical Training Center (TTC)” located in Takasaki, Gunma. The app serves as an efficient education and training tool to equip every engineer with necessary knowledge and skills to develop quality engineers.
Initiated MR tool development toward improvement and standardization of trainees’ levels of understanding
Since about 10 years ago, the NEXCO EAST group has projected the aging of expressways and maintenance and management labor shortage and promoted the Smart Maintenance Highway (SMH) project to match the existing expressway maintenance and management technologies that the group has acquired with ICT technologies. Through the project, operational DX has been advanced with proactive application of digital capacities to the field operations, e.g. conversion of paper-based drawing preparation into tablet-based tasks and analysis and utilization of accumulated data with Microsoft PowerBI. Assigned with the project, Mr. Tetsuo Hideshima, the Managing Director and General Planning Manager of Nexco-East Engineering said with regard to the project policies,
“We at Nexco-East Engineering ensure that drivers can safely use expressways by providing inspections and troubleshooting all day, all year. For continuous maintenance and management of expressways, it is critical to conduct thorough inspections and diagnoses without overlooking any tiny changes in facilities and structures. To that end, streamlining and labor saving based on ICTs are imperative.”
The company also opened TTC in 2012 as a training facility to enhance performance of expressway inspection engineers. For practical training, TTC offers a training environment equipped with real power receiving and distribution equipment, ETC (automatic toll payment system) equipment, and component samples that are exactly the same as those used on expressways.
With regard to the roles of TTC, Mr. Masayuki Namiki, the General Manager of TTC said, “At TTC, over 1,000 of our engineers participate in training programs every year. They go through training to enhance skills based on the internal skill check system to become experts of expressway inspection and maintenance, called ‘highway doctors.'”
Meanwhile, highway structures and facilities have become more and more complex along with the technological advancement and, despite the hands-on learning with such training materials, levels of understanding of the mechanisms and internal structures varied among trainees. This hindered standardization of trainees’ technical levels, constituting an issue for TTC.
The issue was particularly conspicuous in education on ETC equipment. “ETC equipment is composed of multiple devices and how ETC equipment operates was explained with extensive materials such as paper-based radio communication and control signal flow charts, PowerPoint slides, and chalk talks of instructors and then tested with actual ETC devices,” said Mr. Yoshitaka Nakamura, the Manager, Facilities Construction Work Division, Facilities Construction Work Department of Nexco-East Engineering.
“However, it requires considerable time to gain understanding of associations among the training materials. Also, radio communication and control signal flows are impossible to see, so we had to rely on trainees’ imaginations. This resulted in gaps in understanding.”
In response, the company focused attention on the MR (mixed reality) and digital twin technologies. Then a project to develop an “MR for ETC equipment training” was launched out of an idea of combining a 3D model of ETC equipment as well as infrared sensors, radio communication, and data communication routes (which are invisible in reality) with the ETC practice lanes to foster better intuitive understanding.
Endeavor to develop training materials that are easy-to-use for every trainee with no example available as a model
It was DataMesh that contracted with Nexco-East Engineering for its extensive achievements and experiences in the field of digital twins and XR. Mr. Kensho Kashimada, the Director, Marketing and Sales of DataMesh Japan Co., Ltd. looked back, “At that time, there had been no similar attempts at least in Japan. The point was how we could build up MR materials that are easy-to-understand for every trainee while there was no correct answer about what the outcome should be like.”
The first step they took to that end was employment the agile development approach based on a quick PDCA cycles, rather than waterfall-model development that relies on detailed specifications from the beginning. “We carefully proceeded with the development through repeated hearing sessions with the project leading department and TTC instructors who would actually use the tool in practice,” said Mr. Kashimada. They also received the same courses that TTC actually provides to trainees to gain knowledge of ETC equipment mechanisms and applied it to the system.
“By receiving lectures with oral explanations and 2D materials, we witnessed how difficult it is to understand the ETC mechanisms firsthand. We could also appreciate the potential value of the tool we were trying to create. So I think it was a very valuable process,” said Mr. Kashimada.
Thus, after the development period of about six months, accurate 3D models of ETC devices were made and the MR for ETC equipment training using 3D animation superimposed on ETC equipment that is visible via a tablet device or Microsoft MR device “HoloLens 2” was completed.
“Normally, accuracy of SLAM (Simultaneous Localization and Mapping) in positioning of MR superimposition depends on hardware performance. However, for this MR for ETC equipment training, SLAM features of DataMesh’s proprietary digital twin platform (FactVerse) were used to make SLAM accuracy levels of multiple devices uniform,” explained Mr. Kashimada. Ingenuities are made to realize superimposition of massive 3D data without dependence on device performance by using a technology to load necessary portions of data only.
Mr. Kashimada said those features provides trainees immersive experiences in the MR world without stressful events such as misalignment or freezing of 3D animation.
Mr. Namiki said, “It was really a tremendous work to explain invisible things verbally. But now data and radio waves are visible and this increased the level of understanding dramatically.”
Once getting used to the operational feeling of HoloLens 2, trainees can adjust the position just by reading a QR code on the display without taking any complex procedure. It was reported that instructors could acquire basic operation methods only through a one-day lecture provided by DataMesh.
The MR for ETC equipment training, which is well recognized by field operators, too, has been further improved based on feedback from instructors. Currently, additional features such as repeated playback of individual work steps and animation speed adjustment for in-depth explanation make the tool even more effective.
MR for tunnel emergency facilities training making full use of HoloLens 2 characteristics
Out of its confidence in the success of MR for ETC equipment training, Nexco-East Engineering promptly commenced next training MR development, which is the “MR for tunnel emergency facilities training.”
“The internal structure of automatic water sprinkler system for tunnel emergencies cannot be seen from the outside, so we superimposed a CG simulation of the internal water flow and developed training materials for visualizing internals and operating status,” said Mr. Nakamura.
Mr. Namiki said, “In the past, we presented pipe samples from real equipment and illustrations to teach their structures and operations, but we couldn’t actually let water through them to demonstrate the mechanism. Also, explanations somewhat varied among instructors. As with ETC equipment, the gap in levels of understanding among trainees was recognized as an issue.”
As a measure to resolve the issue, DataMesh reproduced the equipment movements from an onset of vehicle fire to watering and visualized the internal structures based on its know-how acquired through development of the MR for ETC equipment training. Furthermore, a feature to adjust transparency of 3D animation of water flowing inside the pipe was added give trainees freedom to observe how water flows and stops inside. Implementation of realistic 3D effects of fire also give realistic experiences to enhance understanding.
Finally, quality of this MR for tunnel emergency facilities training is surprisingly high and precise even to experienced engineers including Mr. Hideshima. “From water flow and the valve opening mechanism to things going on behind pipes are all comprehensible at a glance. I sensed the poser of MR anew,” said Mr. Hideshima.
As the MR for tunnel emergency facilities training is equipped with the “pseudo reproduction feature” to learn points to check in the event of any malfunction and restoration methods, the MR system has a possibility to be applied to a future remote maintenance system.
Plus, linking tablets and other devices has enabled remote training, creating an environment where a greater number of trainees can take part.
Above all, the MR for tunnel emergency facilities training fully taps HoloLens 2 potential compared with the MR for ETC equipment training. In fact, for training using the MR for ETC equipment training with outdoor ETC equipment, tablets are mainly used as output devices because visibility of HoloLens 2 can be deteriorated depending on sunlight intensity. On the other hand, tunnel emergency facilities are indoor equipment and not subject to sunlight. This is where HoloLens 2 comes in.
Mr. Namiki said, “While a tablet needs to be held with both hands, HoloLens 2 is a head-mounted display, which sets both hands free. So trainees can touch devices while going through MR training. It also allows individual trainees to rewind, pause, and adjust the transparency level, giving an advantage in tailoring training sessions to individual levels of understanding.”
Toward even more effective ICTs through professional collaborations
After realizing the quality training tools by using the digital twin and MR technologies, Nexco-East Engineering has a plan to further promote use of ICTs to address the issues of facilities aging and labor shortage, which are expected to be further accelerated.
Exploring possibilities of ICT usages in various scenes, Mr. Nakamura said, “We have an ambition to expand our use of ICTs beyond TTC hands-on training to online training, OJT training, establishment of hands-on safety education system to improve field safety management skills, and remote maintenance, etc.”
With a view to establishment of a system for effective use of ICTs, Mr. Namiki struggling in the field of training said, “In recent years, equipment and facilities have become more and more sophisticated and suffered less failures. It may appear to be a good thing, but on the other side of the coin, most engineers in the future could be inexperienced in failure responses. We would like to establish a system to support engineers with the power of ICTs rather than relying on experienced engineers when the time comes.”
On the other hand, Mr. Hideshima is hopeful about professional partnerships. He said, “Although we have excellent expertise in expressway technologies, we admit we cannot fully catch up with ever-improving ICT seeds and technological innovations. In that regard, we are really hopeful about continuation of partnerships with ICT professionals such as DataMesh and Microsoft Japan.”
Based on ideal partnerships in which expressway professionals and ICT professionals work together to resolve issues and mutually give valuable input to technologies and knowledge, Nexco-East Engineering will continue maintaining and advancing safety on expressways, one of the essentials of our modern lives.
“From water flow and the valve opening mechanism to things going on behind pipes are all comprehensible at a glance. I sensed the poser of MR anew,”
— Mr. Tetsuo Hideshima, Managing Director and Chief of Planning Headquarters, Nexco-East Engineering
“While a tablet needs to be held with both hands, HoloLens 2 is a head-mounted display, which sets both hands free. So trainees can touch devices while going through MR training. It also allows individual trainees to rewind, pause, and adjust the transparency level, giving an advantage in tailoring training sessions to individual levels of understanding,”
— Mr. Masayuki Namiki, General Manager, Technical Training Center Manager, Nexco-East Engineering
“We have an ambition to expand our use of ICTs beyond TTC hands-on training to online training, OJT training, establishment of hands-on safety education system to improve field safety management skills, and remote maintenance, etc.”
— Mr. Yoshitaka Nakamura, Manager, Facilities Construction Work Division, Facilities, Construction Work Department, Nexco-East Engineering
“At that time, there had been no similar attempts at least in Japan. The point was how we could build up MR materials that are easy-to-understand for every trainee while there was no correct answer about what the outcome should be like.”
— Mr. Kensho Kashimada, Director, Marketing and Sales, DataMesh Japan Co., Ltd.
This customer story was originally published by Microsoft here.
