Encore: My Final ASC

On the day ASC24 concluded, I actually felt a profound sense of relief and contentment. This wasn't about the competition results, but rather the realization that I had fully participated in two ASC competitions. I recognized the gap between myself and students from top-tier universities, and I learned far more than just knowledge, skills, and techniques. Most importantly, ASC helped me find my "intuition" for computer architecture. I am deeply grateful and thrilled to have chosen this field—filled with difficulties and challenges, yet utterly captivating—as my research direction.

Logically, the story should have ended there: undergraduate graduation, pursuing a Ph.D., and never crossing paths with ASC again. That was until one day in the Advanced Computer Architecture course, when Professor Shen stopped me with a solemn expression, hoping I would assist in mentoring the 2026 student supercomputing team. I was genuinely surprised. I knew the 2026 team roster was already largely solidified, and I felt quite confident in the lineup. It included returning ASC members Cheng Jingyu and Yu Xiao; Huang Xihan serving as the captain organizing the ISC; and carefully selected students I knew well. For instance, the training monitors for new members Jiang Lan and Zhang Xiaoyu were our former veteran members; Liu Chang had prior ISC experience; Deng Yuzhi was Cheng Jingyu's close friend in the same major; Zhang Han was a guaranteed postgraduate admission candidate; and Cheng Haobo, Qi Hanyu, and He Jiaye were promising lowerclassmen we had discovered. The team drew talent from the general Computer Science category, the Bahe Class, the Qian Class, and the AIBD Class.

To be honest, since my undergraduate days, perhaps influenced by Professor Zhang's subtle yet profound impact, I have harbored a deep interest in teaching. From this perspective, the role transition from a competing team member to a mentoring assistant felt incredibly meaningful. Furthermore, the opportunity to continue challenging the limits of systems on the competition stage alongside familiar peers was a responsibility I was more than willing to embrace. Thus, I happily accepted Professor Shen's invitation.

Throughout the fall semester, we maintained frequent interactions. Professor Shen and I visited Building 3 about once a week. We also organized a sharing session featuring Hu Yu (now a guaranteed postgraduate at Tsinghua University) and Deng Xiangyun (now a direct-track Ph.D. student at Peking University), both veteran members from the ASC22-23 seasons. Our primary goal was to provide foundational learning materials for lowerclassmen, focusing on computer system comprehension and operations. However, we quickly noticed that in today's era of pervasive Large Language Models (LLMs), students habitually turn to LLMs first when encountering problems, or delegate a significant portion of operational tasks to LLM agents. We completely understand and support the extensive use of LLMs and agents, so we did not rigidly demand that team members master every aspect of the command line. An amusing contrast emerged: during my own ASC seasons, competitors were masters of Linux operations, conjuring solutions out of thin air from the command line like hackers in movies. Today, however, the command-line interface has largely transformed into a text box for interacting with programming agents like Claude Code or Codex.

AI is simply too powerful. The preliminary round of ASC featured two tasks: the Embodied World Model (developed by Unitree Robotics) and AMSS-NKCU (a gravitational wave numerical simulation program developed by researchers at National Cheng Kung University, Taiwan). With virtually unlimited access to Claude 4.6 and GPT-5.5 xhigh, our team members could rapidly pinpoint all bottlenecks. Agents directly assisted us in running programs, instrumenting performance profiling, and revealing which functions consumed the most time. For AMSS-NKCU, which is written in Fortran, we rewrote the hot-spot functions in C to facilitate better OpenMP optimization. The preliminary round remained a "black box" as always. Given the varying hardware resources, baseline results, diverse optimization strategies, and the bluffs and truths of progress exchanges among teams, we adhered to our time-honored approach for submitting preliminary technical reports: thoroughly understand the application and our hardware/software systems, analyze performance bottlenecks on our specific platform, propose optimization methods, and test to demonstrate their effectiveness—a classic architecture-oriented software optimization paradigm. For the Embodied World Model, through mixed precision, reusing K/V caches on the GPU, cross-stage video data reuse, and machine learning compiler optimizations tailored to the target architecture, we achieved a 5.27x speedup. For AMSS-NKCU, building on Intel VTune results, we linked it to Intel MKL, applied manual vectorization, optimized memory access patterns, rewrote MPI communications, added our custom eBPF-based thread scheduler, and applied Profile-Guided Optimization (PGO), ultimately achieving a 28.77x speedup. Additionally, we explored a CUDA implementation, porting core functions to the GPU. Since the preliminary test cases were relatively small, the GPU version achieved a 7x speedup. Meanwhile, the machines on the 6th and 7th floors of Room 306 in Building 3 were not yet ready, and the InfiniBand switch speed was limited to 56Gbps, making it impossible to run HPL and HPCG. Just one week before the preliminary deadline, I found an 8-GPU A100 server on vast.ai, where we successfully ran HPL (134.7 TFLOPS) and HPCG (2.221 TFLOPS), ensuring we had something substantial to submit.

The preliminary round was rather unremarkable. However, based on our understanding, preliminary results depend less on the sheer speedup and more on whether the problem-solving methodology is sufficiently novel. We adopted a completely standard software optimization approach: identify the bottleneck, optimize it, and achieve speedup. In the press release issued by the organizing committee after tallying the preliminary results, we noticed that the Fudan University team had transformed this scientific computing application into a tensor computation problem (not an AI model, but rather implementing the physical model algorithm as tensor computation), allowing them to directly utilize modern tensor acceleration libraries like PyTorch. Based on this insight, we decided to continue advancing the GPU version of AMSS-NKCU, porting it to the GPU for every possible computational path. The finals introduced two new applications: QiboTN, a quantum circuit simulation application restricted to CPUs, and LeWorldModel, a newly released world model by Turing Award winner Yann LeCun, restricted to the AMD Radeon W7900D Pro GPU. After careful consideration, we assigned Yu Xiao to tackle QiboTN, and our freshman, Qi Hanyu, to independently drive the LeWorldModel task. During the May Day holiday, we worked for five full days in Room 306-607 of Building 3, making our final preparations for the competition.

For the ASC competition, many settings remain completely unknown until we arrive on-site. QiboTN is a rapidly iterating software, and the officially maintained version is primarily optimized for the CUDA backend, leaving many parts of the CPU backend code as "TODO". In the online introductory video, the organizing committee mentioned severe precision issues (on the order of 10^2) with the MPS MPI implementation. We discovered that during MPS computation, a global canonical center must be maintained. Under the MPI implementation, as the number of qubits increases, different MPI processes maintain multiple global canonical centers (which are not necessarily the true centers), leading to cumulative errors. With the assistance of GPT-5.5 xhigh, we devised a novel approach: without compromising parallelism, we eliminated the need for a global canonical center, thereby circumventing the source of error and improving the weak scalability of the MPS method. This achieved a leap for the MPS method, making it both fast and accurate. I consider this a significant algorithmic innovation. We also successfully ported all computational paths of the AMSS-NKCU application to the GPU, ensuring consistency with the CPU results. In an era without LLMs and agents, it would have been virtually impossible for a single person to accomplish this in just a few months. We spared no effort in securing unlimited token access to the latest models of Claude, Gemini, and GPT through various channels, debugging the equipment, preparing various scripts and tools, and drafting detailed plans and documents to welcome the finals.

To be honest, I am quite seasoned in cluster operations and maintenance. On the first day of setup, we rapidly completed the machine deployment. Just as we had anticipated, the InfiniBand network failed. Following our contingency plan, I began debugging the InfiniBand. The drivers were fine, and opensm was up, but there was a physical link issue. The truth was revealed: out of the four cables provided by the organizing committee, only two were functional. Jingyu retrieved new cables from the committee, and we began debugging the applications. The machine time allocation on the first day encountered some issues; every application wanted to run, but we were debugging while running, so we didn't strictly follow the plan. This year, the CPUs were switched from Intel to AMD. The EPYC 9755 uses a Chiplet architecture, with every 8 physical cores arranged on a single CCD. Furthermore, the AMD CPU architecture possesses certain mysterious optimizations (for instance, certain memory access patterns described in our classic architecture textbooks as unfriendly to processors indeed cause performance degradation on Intel CPUs, but are barely affected on AMD CPUs—I'm not sure what kind of "black magic" AMD employs). Whether adjusting certain options in the BIOS or tuning cache and core affinity configurations on the software side, things felt somewhat elusive. To reduce base power consumption, we first changed the BIOS setting from "Custom" to "Balanced" and rebooted. At this point, we discovered that the CPU version of HPL would not run, constantly prompting that the core count was not correctly bound. With the help of an LLM, Xihan began troubleshooting whether a specific mpirun parameter was misconfigured or if Intel OneAPI's mpirun had some specific implementation causing the error. After half an hour of tense troubleshooting, Xihan casually executed the lscpu command, and we suddenly realized that the 256 cores of the dual CPUs on each node had magically shrunk to 224 cores—a loss of 32 cores! I suddenly realized that this meant one core was missing from each CCD. I panicked slightly, as I couldn't remember what exactly needed to be changed in "Custom" mode to roll back. Then, I recalled that disabling IOMMU could cause certain cores to be lost, and it immediately clicked: the "Balanced" mode had disabled IOMMU, causing one core within each CCD to assume the role of the MMU. However, there was no way to revert to the previous "Custom" mode. So, I swiftly reset the BIOS, provided a new configuration myself, disabled unnecessary CPU options and I/O ports, and made some fine-tuning to certain CPU behaviors. Ultimately, I returned to a familiar environment, reducing the idle power consumption of the four servers and one InfiniBand switch from 2400W to 1180W.

At this point, the machines were running smoothly. After a brief meeting, we allocated the machine time for the next three hours, made minor tweaks to HPL and HPCG, and ensured that the scripts and environment variables remained stable, ready to run at any moment during the official competition. On the evening of the first setup day, we held the first mock defense for the ISC competition. The back-to-back nature of the two competitions began to make me feel the pressure.

On the second day of setup, addressing new issues with the on-site cluster, we managed to capture the server BMC API through a hack and, with the assistance of an Agent, completed a fan control script. Initially, we controlled the server BMC via the web interface, but we noticed that even when we fixed the fan speed to a specific RPM, the server would still intelligently adjust it, not fully adhering to our settings. The Agent helped us write a script that sent a fan adjustment request to the BMC every 5 seconds, effectively locking the fan speed via polling. Immediately after, we realized that the AMD CCD architecture dictated that we could not simply disable certain cores or set different performance modes for different cores as we could with Intel CPUs. Observing this, we on-the-fly tuned a configurable power manager. When running different tasks, it could dynamically adjust to a specific performance mode. Based on the CPU's TDP, when the CPU hit our set power wall, it would automatically lower its frequency to return to a safe power envelope. For the team application ICON, we quickly scanned the runtime parallel parameter configurations and found what we believed to be the optimal solution.

In terms of system setup, we have consistently been at the forefront of system stability, reliability, and power consumption control. The time taken from racking the four servers to installing all software packages, dependency libraries, and applications did not exceed one hour. This has always been our advantage, built upon a wealth of accumulated experience. Even when faced with the new servers provided by the committee in ASC24 (which left most teams unable to control power consumption), or this year's shift to AMD CPUs featuring a heterogeneous architecture of AMD CPUs, AMD GPUs, and NVIDIA GPUs, we were still able to rapidly complete the system setup. Meanwhile, most other teams were more or less stuck on certain issues. Our system operations and maintenance were highly recognized by the organizing committee; we were awarded the "Reliable Cluster Award" right at the beginning of the first day of the official competition. On the second day of setup, everyone strictly adhered to the scheduled machine time for debugging, and everything proceeded quite smoothly. In the afternoon, the ISC team went to the classroom to continue polishing their online presentation.

Then came the first day of the official competition. Jingyu and I maintained a phone connection between the inside and outside of the venue, and we quickly obtained the HPL results. Since we only had four A100 GPUs, we didn't harbor high hopes for HPL and HPCG. At 8:30, we finished the HPL test and began running HPCG. On the other two machines without A100 GPUs, we started debugging ICON. Half an hour later, HPCG finished, and we began running the Embodied World Model. Then, something bizarre happened: our optimized Embodied World Model failed the correctness validation on the final test cases! We immediately decided to revert to the baseline version, only to find that the baseline version also failed the correctness validation! I could distinctly feel the panic setting in among our team members, both inside and outside the venue. At 10:50, we ported the official reference run guide to our machines, fixed a bug, and finally got ICON running. Fortunately, although this was the largest test case, we had correctly guessed this specific case during our parameter scanning the day before, so our estimation of its runtime was highly accurate. Xihan called Deng Yuzhi out of the venue to debug the Embodied World Model together. After an hour of debugging, we finally discovered that one configuration was incorrect. In our post-mortem, we found that according to the committee's manual, only one specific configuration should be modified. However, almost all LLMs would bind this configuration with another, treating them as the same. When using an Agent, it would modify both, and that extra, unwarranted modification directly caused the result's precision to fail the requirements. Because debugging ICON and the Embodied World Model took far longer than expected, our machine time schedule for the day was completely derailed. While waiting for ICON to finish at 13:30, the Embodied World Model was still not debugged. At this point, we discovered that the second test case for ICON also had a bug, requiring more on-site debugging and code modification. During this process, I was growing increasingly anxious, especially watching our team member throw the runtime logs into the Claude Code dialog box, and then just watch as Claude Code began reading the code, executing commands, and initiating various troubleshoots... The more I watched, the more anxious I became. I realized our team member was visibly panicking, their brain seemingly crashing, acting like an intelligent zombie merely pasting problems to the Agent, which then began its long, drawn-out response. The Embodied World Model was finally debugged at 14:00, and we spent about 40 minutes running the baseline code. Meanwhile, ICON was still being debugged. Suddenly, we noticed all CPUs spiking. Looking closer, we realized Claude Code had figured out the error and got the application running.

At that moment, we should have immediately stopped ICON, examined exactly what commands Claude Code had executed, and rerun it ourselves. But we didn't. As mentioned earlier, the team member responsible for this task was completely unable to think clearly, had no idea what Claude Code had configured in the background, and was uncertain if the test case could be restarted once killed. I then realized our team member had become entirely dependent on the Agent. During preparation, the Agent had even written a script to help us monitor ICON's progress via runtime logs. However, for the second case, it was unclear how it was launched, and the progress-monitoring script yielded no output, leaving our member utterly helpless. I was anxiously watching from the side, saying, "Just tail the log file and use watch -n 5 to refresh!" Only then did the member snap out of it. Looking back, there were several points where we should have stopped. If we didn't know how a task was launched, we should have stopped. If the progress script stopped outputting after it started running, we should have stopped. But we didn't heed the warning signs. Clinging to a sliver of hope, we let it run for another hour and a half, only to find it failed the correctness validation. This threw us into even greater disarray. On the Embodied World Model side, Xihan and Deng Yuzhi were debugging in a fog; the code versions were so chaotic that our "stitching together" strategy completely failed. We were thus trapped in endless debugging. At this point, we heard that Peking University's ICON model could finish the smallest test case in just 30 minutes. We immediately started ours, but the progress bar indicated it would take 7 hours! The situation was critical. We frantically adjusted parallel parameters and finally brought it down to 144 minutes by 5:00 PM, but it was already far too late. Consequently, the Embodied World Model also failed to produce a version that could pass the correctness validation. Amidst the tension and chaos, the first day of the competition abruptly came to an end.

After the first day, I could sense a palpable sense of dejection among everyone. I was also under immense stress. After the competition hours ended, almost no one had an appetite for dinner. We held our daily debriefing right there in the spectator seats outside the venue. The meeting summarized the day's lessons learned and allocated the machine time schedule for the following day.

The second day proceeded relatively smoothly. AMSS-NKCU ultimately only required one computational path, which we had already fully ported to the GPU during the preparation phase. We completed the first test case in 170 seconds (the baseline took over 20 minutes), and swiftly ran the remaining two. Immediately after, LeWorldModel finished running the baseline code and secured the correctness score. We then considered further optimizations on-site, implemented early stopping, and ultimately reduced the runtime to 150 seconds (the baseline was around 30 minutes). The greatest challenge lay with QiboTN. At the finals venue, we were given an entirely new codebase, a specific version mandated by the committee, with a forced requirement to use a particular backend. Fortunately, we managed to get it running, replaced the backend computation library, and executed all test cases. At this point, we encountered a dilemma: we didn't know what the exact correct answer was. We only knew that our results matched Zhejiang University's perfectly, and Shanghai University's matched Taiwan University's perfectly, but ours differed from theirs. Our members inside the venue panicked again, insisting on rerunning the test case, and even wrote down two different answers on the answer sheet (which led to the committee calling us that night to ask why we provided an answer for the second one without the corresponding runtime logs)... Then, we heard that Beijing University of Posts and Telecommunications could complete our 1.5-hour task in just over 10 seconds, because they had deleted the circuit construction part of the quantum circuit simulation (which consumed 99.9% of the time), yet astonishingly found that it did not affect the final result.

We considered this outcome unacceptable and reported the situation to the organizing committee, but our concerns were not adopted. After the competition ended on the second afternoon, we went to print our poster and then conducted the final rehearsal for the ISC online defense.

During the poster presentation session, to our pleasant surprise, our poster actually secured second place. According to Xihan, Professor Torsten Hoefler even gave us a perfect score. Regrettably, we ultimately placed 10th overall. The main issue is likely that nowadays, every competition task sees ultra-high speedups. Furthermore, there is still a difference between a competition and actually running programs in a real supercomputing center. Time-limited competitions focus more on how to obtain compliant results within the stipulated time, and we often failed to focus enough on this competitive aspect. It is evident that everyone tried their absolute best. However, there are always some regrets every year. Perhaps we have not quite kept pace with the changing world; thus, we step into new pitfalls every year, while schools that have adapted to these changes and trends can adjust much more seamlessly.

How to collaborate effectively with LLM Agents is also a new challenge. While Agents can vastly enhance individual capabilities, from a macro perspective, we can easily espouse the generic notion that "machines are merely aids, tools are just references, and humans must make the final call." Such viewpoints are easy to articulate. But the crux of the matter is: how should we constrain the Agent, and more importantly, how can we maintain a clear head under extreme time pressure, rather than surrendering ourselves entirely to the Agent? From this ASC experience, two things are clear. First, unlike programming contests like ACM/ICPC, or CTF competitions with clear rules and standard answers, ASC cannot yet be completely replaced by LLMs and Agents (whereas in programming and CTF contests, humans can no longer outperform AI). Second, given that everyone is using the world's most advanced AI, it is worth pondering why some schools still perform significantly better. What AI lacks is a holistic understanding of specific hardware and software systems, which is precisely the core of software optimization. Our human brains, generally considered to possess a sufficiently long context window, are thus capable of making more rational decisions. Therefore, no matter what, the fundamental skills of team members must be valued, and we must simultaneously inspire them to proactively explore wildly creative ideas. Fundamental skills are the foundation of these ideas, and beyond that foundation, ideas require broader vision and the guidance of disciplinary thinking.

As an undergraduate cultivated by this very college, I must objectively evaluate that, based on my experience, the current undergraduate training model struggles to produce such students. I believe many students are actually quite capable and hardworking. Some students at local universities do not endure the ascetic study life we do here, yet due to better resources and broader horizons, their career prospects are excellent. Regarding the construction of the Bahe Class, my impression is that no one has truly cared about it in a practical sense; too much effort has been spent on superficial, abstract concepts, while very few things can be genuinely implemented. I remember on the day I registered as an undergraduate, Professor Mao Xiaoguang told me, "Studying and living here might make you feel constrained, with everything pushed along a predetermined track, but your mind must never follow blindly." The Bahe Class (including the general Computer Science category) recruited a batch of students with excellent college entrance exam scores, yet their career outcomes have not been ideal. Meanwhile, peers with similar exam scores who attended local universities have managed to find good jobs or more easily advance to better institutions for further studies. I have served as a teaching assistant for "University Computing (Part II)" for the junior students in the local Computer Science categories of the classes of '24 and '25. Whenever standard questions are given, they always score very high. However, once the questions vary even slightly, they become completely lost. I believe this is exactly the same as our helplessness when encountering unexpected situations on the ASC stage: everyone's vision and thinking are too narrowly confined. This reminds me of my own undergraduate days, when I was sent on official duties to attend various reports by outstanding students of the university. I always felt that it was incredibly difficult for them to cast their gaze beyond the walls of Building 3.

Now, on the train back to Changsha, I feel as though I have graduated from undergraduate studies all over again. Frankly speaking, I have always received immense care and support from the college, and I sincerely hope the college continues to thrive. However, I firmly believe that if we do not implement radical reforms in how we cultivate our students, we may truly fall behind the times.