Chair of Business Informatics
Saarland University

Streamlining LLMs: Adaptive Knowledge Distillation for Tailored Language Models

Prajvi Saxena, Sabine Janzen, Wolfgang Maass:

Large language models (LLMs) like GPT-4 and LLaMA-3 offer transformative potential across industries, e.g., enhancing customer service, revolutionizing medical diagnostics, or identifying crises in news articles. However, deploying LLMs faces challenges such as limited training data, high computational costs, and issues with transparency and explainability. Our research focuses on distilling compact, parameter-efficient tailored language models (TLMs) from LLMs for domain-specific tasks with comparable performance. Current approaches like knowledge distillation, fine-tuning, and model parallelism address computational efficiency but lack hybrid strategies to balance efficiency, adaptability, and accuracy. We present ANON - an adaptive knowledge distillation framework integrating knowledge distillation with adapters to generate computationally efficient TLMs without relying on labeled datasets. ANON uses cross-entropy loss to transfer knowledge from the teacher's outputs and internal representations while employing adaptive prompt engineering and a progressive distillation strategy for phased knowledge transfer. We evaluated ANON's performance in the crisis domain, where accuracy is critical and labeled data is scarce. Experiments showed that ANON outperforms recent approaches of knowledge distillation, both in terms of the resulting TLM performance and in reducing the computational costs for training and maintaining accuracy compared to LLMs for domain-specific applications.

Energy-efficient artificial intelligence for sustainable data centers

The research project ESCADE is presented at Hannover Messe 2025 from March 31 to April 4 at the booth of the Federal Ministry for Economics and Climate Change (Hall 2, Booth A18).

The climate neutrality goals, rising energy costs and the limitations of traditional hardware are jeopardizing the economic viability of German data centers, which are vital for the digital transformation using AI. Their massive energy consumption (16 billion kWh in 2020) is set to be reduced through the use of neuromorphic hardware and energy-efficient AI algorithms.

ESCADE optimizes the energy efficiency of AI in data centers. Approaches such as knowledge distillation, quantization and neural architecture search result in smaller, more energy-efficient AI models that nevertheless deliver a comparable performance. When combined with neuromorphic hardware, these models can achieve energy savings of up to 90%.

An ESCADE AI agent optimizes the energy efficiency of algorithms throughout their development, training and deployment. By making data-driven decisions, it enables resource-efficient solutions, for example in scrap sorting and software development, without performance loss. Small and medium-sized enterprises, in particular, benefit from AI that is both economically and environmentally sustainable.

The ESCADE exhibit visualizes the project results in the form of a physical table demonstrator. The technical prototype EAVE - Energy Analytics for Cost-Effective & Sustainable AI Operations empowers decision-makers with real-time insights into the energy efficiency of AI workloads. The tool supports measuring, predicting and optimizing operational costs, energy-efficiency and CO2 emissions of AI models including the optimization of data center power usage effectiveness (PUE). The EAVE measurement module analyzes current cost- and energy-efficiency, while the prediction module enables analysis of future AI workloads with respect to cost and energy. The optimization module analyses in-depth AI model optimization, focusing on compression techniques for Large Language and Vision Models in the context of industry use cases.

ESCADE is funded from 01.05.2023 - 30.04.2026 by Federal Ministry for Economics and Climate Change (BMWK). The research team of Prof. Maaß at German Research Center for Artificial Intelligence (DFKI) is coordinating the project and collaborates with the funded partners NT Neue Technologie AG, Stahl-Holding-Saar GmbH & Co. KGaA, SEITEC GmbH, Technical University Dresden, University Bielefeld and the Austrian Partner Salzburg Research.

Advancing Quantum Computing for Manufacturing Simulation

The final meeting of the research project QUASIM: QUANTUM COMPUTING ENHANCED SERVICE ECOSYSTEM FOR SIMULATION IN MANUFACTURING took place at the German Research Center for Artificial Intelligence (DFKI) in Saarbrücken. This collaborative project aimed to explore how quantum computing can enhance simulation technologies in the manufacturing sector, particularly within the metalworking industry.

With over 390,000 companies and approximately 3.7 million employees, the metalworking industry represents the largest secondary sector within the EU-28 (Eurostat, Sectoral Analysis of Key Indicators). Machining is one of the most significant manufacturing technologies in this sector, crucial for industries such as tool and mould making, the semiconductor industry, and engine construction. Given the industry's reliance on machining, companies are continuously striving for higher quality, productivity, cost-effectiveness, and sustainability.

Quantum Computing for Manufacturing Optimization

State-of-the-art manufacturing processes can be optimized with the support of computer simulations, which are often time-consuming and expensive. QUASIM focuses on mathematical problems that frequently arise in industrial simulations, developing quantum algorithms that can solve these problems faster or with less computational storage than classical approaches. The research aims to improve scalability based on problem-dependent parameters, such as the number of equations to solve or the desired accuracy.

By considering large-scale mathematical problems common in manufacturing simulations, QUASIM aims to revolutionize computational efficiency and industrial applicability.

Use Cases of Quantum Computing in Manufacturing

Use Case 1: Milling Process Optimization

Milling is a metal-cutting manufacturing process where a multi-toothed tool rotates to create various workpiece surfaces. Unlike other processes such as turning or drilling, milling features constant cutting interruptions, which can cause dynamic excitations leading to vibration marks on the workpiece surface. To address this issue, dynamic process stability simulations are performed to analyze vibrations and optimize milling process design, particularly for thin-walled components.

In QUASIM, a dexel-based engagement simulation is followed by a finite element method (FEM) simulation workflow, involving:

- IPW (in-process workpiece) conversion from dexel to solid

- Meshing process

- Modal analysis

A key challenge is the solution of eigenvalue problems in modal analysis, which are computationally intensive. Quantum algorithms are being investigated to accelerate eigenvalue problem solving, potentially reducing simulation time and improving accuracy in milling process optimization.

Use Case 2: Optimizing Laser Cutting Through Thermal Expansion Analysis

In laser cutting, accurately predicting and managing thermal expansion in metal sheets is critical to ensuring precision and preventing defects. The heat generated by the laser causes thermal deformation, which affects the final shape and quality of the manufactured component. Simulations are essential for modeling these effects, but traditional finite element method (FEM)-based simulations are computationally expensive.

QUASIM investigates quantum-enhanced approaches for thermal expansion analysis, focusing on solving large-scale differential equations governing heat transfer and deformation. Quantum algorithms are being explored to:

- Accelerate the solution of heat diffusion equations

- Improve the accuracy of temperature distribution predictions

- Enhance optimization of laser parameters for high-precision cutting

By leveraging quantum machine learning (QML) techniques, the project aims to develop faster and more accurate thermal simulations, enabling manufacturers to reduce waste, increase precision, and improve energy efficiency in laser-based manufacturing processes.

Quantum Computing as a Game Changer

Initial research within QUASIM suggests that quantum computing offers significant advantages in addressing these challenges. Quantum mechanical principles have the potential to:

- Accelerate numerical simulations dramatically

- Improve simulation accuracy through quantum machine learning (QML)

- Enable real-time optimization of machining processes

These findings indicate that quantum-enhanced computing could revolutionize manufacturing by making high-fidelity simulations feasible for industrial use.

Participants and Key Stakeholders

The final meeting brought together key stakeholders from academia, industry, and government, reflecting the project's broad impact and interdisciplinary nature. Notable participants included:

- Wolfgang Förster, Staatssekretär des Finanz- und Wissenschaftsministeriums des Saarlandes

- Dr. Glasmacher (BMWK)

- Dr. Grass (DLR)

- Prof. Frank Wilhelm-Mauch (FZJ)

- Prof. Wolfgang Maaß (DFKI, Project Coordinator)

- Dr. Tobias Stollenwerk (FZJ)

- Dr. Valentina König (moduleworks)

- Sven Danz (FZJ)

- Alejandro Delgadillo (moduleworks)

- Marco Kulig (TRUMPF)

- Rivan Ruguhbar (FZJ)

- Anika Rusch (DFKI)

- Stefan Schröder (Fraunhofer IPT)

- Nirav Shinoy (DFKI)

- Hannah Stein (DFKI)

The meeting marked the culmination of years of research and innovation, demonstrating the potential of quantum computing to transform manufacturing simulation. As the industry moves forward, continued collaboration between research institutions, technology providers, and manufacturers will be essential in bringing these advancements from the lab to real-world applications.