Emptying the mind, one article at a time

One of the most common misconceptions in AI engineering is that you always need a Python runtime to execute models.

This is where ONNX (Open Neural Network Exchange) is critical. In 2017, Microsoft and Facebook realized they had a problem: framework Lock-in.

At the time, if you trained a model in PyTorch, you were stuck there. Deploying it to production often meant rewriting code or using slow wrappers.

Their goal was to create a “Universal Interchange Format” a standard that allowed models to be trained in flexible frameworks (like PyTorch) but run on high-performance inference engines (like ONNX Runtime) without being tied to the original training environment.

Google did not join the ONNX partnership initially; they had their own ecosystem but over time, it did become the standard bridge between PyTorch and TensorFlow,

In the modern AI landscape, 99% of model development happens in one of two places: PyTorch or TensorFlow.

Problem

Usually, running AI in Java means creating a sidecar Python service or using slow HTTP bridges.

If you are a Java shop building AI features, you might go through the following steps:

  1. Data Science team builds a model in PyTorch/TensorFlow.

  2. Engineering team has to wrap it in a Flask/FastAPI container.

  3. You end up managing two languages, two CI/CD pipelines, and a massive Python runtime (often 3GB+) just to perform a simple calculation.

Only recently (with the maturation of LangChain4j and better Java bindings for ONNX Runtime, it has become a viable way to replace Python entirely in enterprise backends.

ONNX is like the “PDF” for machine learning models.

  • Python (PyTorch/TensorFlow): It’s the editor where you create, train, and tweak the model. It’s heavy and complex.

  • ONNX: Is the exported, static artifact. It serializes the model into a computation graph (a set of nodes and edges representing mathematical operations).

The Runtime Architecture

When this system runs, it doesn’t spin up a hidden Python process or make HTTP calls to a flask server. It uses the Microsoft ONNX Runtime (ORT).

  • ORT is a high-performance inference engine written in C++.

  • The Java application communicates with ORT via the Java Native Interface (JNI).

  • This allows us to run the model on the CPU (using AVX2/AVX512 instructions) or GPU directly from Java, often faster than the original Python implementation because we bypass the Python Global Interpreter Lock (GIL).

Result: you get near-metal performance with zero Python interpreter overhead, no CPython Global Interpreter Lock (GIL), and no pip install nightmares in production and therefore you can full take advantage of Java multithreading capabilities. Java has no GIL: If your server receives 100 requests to vectorize documents, Java can utilize all cores of the server simultaneously to process them. By using Java, you unlock the hardware’s full potential without complex workarounds (like multiprocessing).

The Tokenization Challenge

The trickiest part of “Python-free” RAG isn’t the model it’s the tokenization (converting text “Hello” into numbers [142, 7489]).

Even if you have the model in ONNX, you still need Python to perform tokenization

In Python, we take transformers.AutoTokenizer for granted, but under the hood, it is performing a complex process:

  1. Normalization: Unicode formatting (NFC vs NFD), lowercasing, and stripping accents.

  2. Pre-tokenization: splitting text by whitespace or punctuation (e.g., “don’t” -> “don”, “‘t”).

  3. Model Mapping: applying algorithms like BPE (Byte-Pair Encoding used by GPT-4) or WordPiece (used by BERT) to merge characters into sub-word tokens and therefore understand the concept of the word by analyzing its parts, even if it has never seen the full word before.

In Java, LangChain4j and the underlying DJL/ONNX dependencies handles this natively It reads the standard tokenizer.json file (exported from Hugging Face) for a Model and performs the text-to-ID conversion entirely in Java before feeding the tensors to ONNX.

The Pure Java Pipeline:

  1. Input: String (Java)

  2. Tokenization: Native Java implementation (No Python)

  3. Inference: ONNX Runtime (C++ via JNI)

  4. Output: Vector Embedding (Java float array)

This architecture is what allows mcp_server4j to run as a single, self-contained JAR file with zero external dependencies.

Updated: