LFX Mentorship: Harnessing WasmEdge and Runwasi for Cloud-Native WASM Services

This article chronicles a significant LFX Mentorship journey undertaken by @vatsalkeshav, delving into the powerful synergy of WebAssembly (WASM) and Kubernetes orchestration. This narrative aims to provide valuable insights for anyone navigating the cloud-native landscape.

Table of Contents

• The Pervasive Influence of WASM
• The Collaborative Endeavor
• Navigating the Initial Complexities
• Conquering Kubernetes Deployment
• Scaling to Multi-Node Architectures
• Ensuring Stability with Automated Tests
• Profound Project Discoveries
• Future Directions and Unfinished Chapters
• Acknowledgements

The Pervasive Influence of WASM

Every impactful story has a beginning, and for this project, it was WebAssembly (WASM) – a versatile, secure, and cross-platform binary instruction format. While commonly associated with client-side browser computing, WASM’s utility extends far beyond, encompassing crucial non-exhaustive applications such as edge computing, portable AI at the edge, and even the potential to revolutionize traditional containerization.

• Client-Side Computing: By leveraging the user’s local computing resources instead of server processing power, WASM significantly reduces server costs and enhances user experience with near-native performance for applications ranging from PDF tools and IDEs to high-end games directly within the browser.
• Edge Computing: Similar to client-side applications, edge computing processes data closer to the source (e.g., routers, network towers, traffic cameras), minimizing latency and bandwidth usage.
• Portable AI on the Edge: WASM enables the deployment of AI models on lightweight WASM servers, some as small as 12MB (e.g., LlamaEdge’s llama-api-server), often implemented efficiently in Rust.
• WASM Containers: Representing a monumental technological leap, WASM containers offer sandboxed environments for WASM binaries. When traditional container technologies like Docker are superseded by runwasi’s WasmEdge container runtime, image sizes, build times, and container boot-up times can be reduced by up to 90%.

(It’s worth noting that the full scope of WASM’s capabilities was a discovery made during this project, building upon existing Rust knowledge.)

The Collaborative Endeavor

The journey began through the LFX mentorship portal. Armed with Rust expertise, and guided by mentors Vincent Lin and Yi Huang, the project aimed to enhance Runwasi. As WasmEdge, implemented in C++, evolved as a key runtime for Runwasi, a critical need emerged for robust verification processes to ensure stability across container and cloud environments.

The project’s key objectives included:

1. Thorough research into relevant codebases, tools, and environmental setup.
2. Verifying system behavior within Kubernetes (k8s) clusters.
3. Clarifying plugin system configurations in k8s, including environment variables and dynamic library loading paths.
4. Establishing a CI repository for integration verification and result demonstration.
5. Innovatively integrating HTTP services and the plugin system in a multi-node setup to showcase cloud deployment efficacy.

Navigating the Initial Complexities

The initial phase involved navigating the intricate, albeit well-documented, WASM ecosystem. This foundational learning took nearly three weeks before tangible progress was achieved. Starting with fundamental steps, the project involved deploying simple WASM applications like ghcr.io/containerd/runwasi/wasi-demo-app:latest as WASM pods in Kind and k3s environments. This period marked the introduction to Kubernetes pods and WASM-specific pods, with k3s proving to be an invaluable tool.

During this time, OCI runtimes such as crun and various Runwasi shims (WasmEdge, Wasmer, Wasmtime) were explored. A significant milestone was successfully replacing k3s’s bundled containerd’s OCI runtime, runc, with crun. This challenging task, requiring four days and nights, underscored the principle that momentum, once gained, propels further success.

Conquering Kubernetes Deployment

The next phase involved deploying LlamaEdge’s llama-api-server within Kubernetes. A significant hurdle was encountered when pods repeatedly entered a container restarting state. Diagnostic steps included reviewing pod logs and disassembling the WASM binary. The breakthrough came from mentor Vincent Lin’s experience-driven hypothesis: the plugin dynamic library might not be correctly detected by Runwasi within k8s.

This led to the meticulous mounting of all /.wasmedge/plugin/libwasmedgePluginWasiNN.so dependencies into the container. This resolution was instrumental in the project’s successful mid-term evaluation. Furthermore, GitHub Actions were integrated to establish a daily CI pipeline for continuous verification. This work is expected to contribute to the official LlamaEdge documentation on Kubernetes deployments.

Scaling to Multi-Node Architectures

The project then escalated to integrating an HTTP service and the WASI-NN plugin system within a multi-node Kubernetes environment. A prototype, multi-wasm-pod-demo, was developed in Rust, showcasing a load-balancer (also a WASM pod) managing multiple LlamaEdge llama-api-server WASM pods, each running different GGUF models. A non-WASM pod, acting as a service-watcher, utilized the kube-rs client for dynamic service management.

An initial challenge arose with WASM pods (using Runwasi’s WasmEdge shim) failing to resolve DNS from service names. To overcome this, the non-WASM service-watcher pod, leveraging kube-rs, was employed to ensure dynamic service discovery. Automated CI workflows were also implemented for this multi-node setup. Subsequent guidance revealed that the DNS resolution issue was addressable by using the latest tokio_wasi crate and configuring the DNS_SERVER environment variable in the deployment YAML, once again highlighting the value of experienced insight.

Ensuring Stability with Automated Tests

To guarantee the reliability of the deployed services, robust pod tests were incorporated into the CI pipelines of both runwasi-wasmedge-demo and load-bal-llamaedge-demo, facilitated by Bash scripting. These tests included:

• Pre and post-request pod health checks, generating comprehensive reports on pod status, container readiness, restarts, events, and resource utilization.
• Service health checks, logging endpoints, service status, and service information.
• For load-bal-llamaedge-demo, detailed load-balancer pod logs were collected and segmented by event markers, providing granular insights into its operation.

Profound Project Discoveries

This mentorship journey proved to be exceptionally rewarding, unveiling the immense potential of WASM in real-world cloud deployments. The project demonstrated how WASM workloads can effectively supersede traditional container and VM approaches.

Two of the decade’s most significant technological advancements were directly experienced:
* LlamaEdge’s capability to develop AI servers that run efficiently at the edge, leveraging Rust.
* WasmEdge’s role in replacing conventional containers with highly optimized WASM alternatives.

This mentorship provided a unique opportunity to learn and contribute to these groundbreaking technologies.

Future Directions and Unfinished Chapters

While substantial progress was made, avenues for future development remain. The service-watcher in load-bal-llamaedge-demo currently operates as a non-WASM pod. This is due to its dependencies, including kube-rs and k8s-opensapi, which rely on a stack (reqwest → hyper → tokio → socket2) that assumes native sockets, threads, and system TLS. Although WasmEdge offers compatible forks like tokio-wasi, reqwest-wasi, hyper-wasi, and socket, kube-rs maintains hard dependencies on native crates, necessitating patching kube-rs itself for WASM compilation. For now, running the watcher as a traditional pod is pragmatic, but compiling the service-watcher to the wasm32-wasip1 target remains a desirable future goal.

Acknowledgements

My deepest gratitude goes to my mentors, Vincent Lin and Yi Huang, whose invaluable guidance and unwavering support were instrumental in my growth and the project’s success.

I am also thankful for the opportunity to engage with and receive assistance from the vibrant WasmEdge and Runwasi communities. Special thanks to @hydai for crucial assistance with DNS resolution, and to @steel-bucket for his helpful development contributions and friendship.

Above all, sincere appreciation to the Linux Foundation and CNCF for making this mentorship program possible.