Intelligent Predictive Maintenance RAG Framework for Power Plants: Enhancing QA with StyleDFS and Domain Specific Instruction Tuning

Abstract

Process plants are complex large-scale industrial facilities that convert raw materials or intermediate products into final products, requiring continuous processes with high safety and efficiency standards. In particular, in nuclear process plants, Predictive Maintenance System (PMS) plays a critical role in predicting equipment anomalies and performing preventive maintenance. However, current PMS relies heavily on the experience of a few experts, leading to knowledge loss upon their retirement and difficulty in swift response. Existing off-premise Question-Answering (QA) systems based on Large Language Models (LLM) face issues such as data leakage and challenges in domain-specific tuning. To address these problems, this study proposes an on-premise intelligent PMS framework utilizing a new chunking method, StyleDFS, which effectively reflects the structural information of documents. Additionally, we demonstrate that Instruction tuning using relevant domain-specific data improves LLM performance even under limited data conditions.

Key Contribution

Domain-specific RAG framework for scientific and industrial QA; led to two technology transfers.

Method

The framework targets predictive maintenance in nuclear process plants, where maintenance knowledge lives in long structured documents (mainly HWP and DOCX) and must stay on-premise.

• StyleDFS chunking. Documents are converted to HTML while preserving their structure. The style classes that the converter assigns to each HTML element define a tree in which titles, body text, subsections, and lists become separate nodes. A preorder depth-first search then accumulates text from leaf nodes and their parents; when the accumulated text exceeds the maximum length, the chunk is closed and a new one starts from the right sibling, keeping the parent node's text so that each chunk retains its section context.
• Retrieval. Among Korean-capable embedding models from the MTEB leaderboard, multilingual-e5-large gives the best nDCG@3 on Ko-StrategyQA, Ko-mrtydi, and Ko-miracl and is used for retrieval. Chunk embeddings are stored in PostgreSQL and served with Text Embedding Inference (TEI) so that a single GPU can handle multiple users.
• Domain-specific instruction tuning. MRC data from science and technology documents is rewritten into descriptive answers with GPT-4 and used to instruction-tune Llama-3-8B, Llama-3-Open-Ko-8B, gemma-7b, and gemma-ko-7b.

Results

Embedding model selection (nDCG@3).

Retrieval recall by chunking method across n-gram lengths, at top-1 and top-3.

Answer generation (BLEU and ROUGE-L) for base and instruction-tuned models. Gold uses chunks that contain the correct answer.

• StyleDFS gives the highest retrieval recall in every setting, for example 0.3595 (top-1) and 0.5710 (top-3) for 1-gram, compared with 0.2464 and 0.4363 for semantic chunking.
• Averaged over the four models, StyleDFS gives the highest BLEU and ROUGE-L among the three chunking methods in both the top-1 and top-3 settings; answers generated from gold chunks remain higher.
• The instruction-tuned model scores higher than its base model in 51 of the 56 base/tune pairs in the generation table.

BibTeX

@inproceedings{hong-etal-2024-intelligent,
    title = "Intelligent Predictive Maintenance {RAG} framework for Power Plants: Enhancing {QA} with {S}tyle{DFS} and Domain Specific Instruction Tuning",
    author = "Hong, Seongtae  and
      Shin, Joong Min  and
      Seo, Jaehyung  and
      Lee, Taemin  and
      Park, Jeongbae  and
      Young, Cho Man  and
      Choi, Byeongho  and
      Lim, Heuiseok",
    editor = "Dernoncourt, Franck  and
      Preo{\c{t}}iuc-Pietro, Daniel  and
      Shimorina, Anastasia",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2024",
    address = "Miami, Florida, US",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-industry.61/",
    doi = "10.18653/v1/2024.emnlp-industry.61",
    pages = "805--820"
}