In this news:
On 2026-08-14 I published
an article on the Google Security blog
with an update on HEIR, our homomorphic encryption (HE) compiler.
This is a companion article,
in which I have no limits on word count or jargon,
and I can feel free to be honest.
So strap in.
Assuming you won’t read the linked corporate blog post,
HEIR is a compiler that converts an input program to
a program that operates directly on encrypted data.
The guarantee of homomorphic encryption is that,
assuming you haven’t cracked the cryptography,
at no point does the computer running the program
get even a single bit of information about the cleartext data
used to generate the encrypted inputs.
No information about the inputs, outputs,
or any intermediate values.1
The blog post focuses on HEIR’s ability
to compile pre-trained ML models,
and gives four examples of small, but nontrivial models
that it can compile.
Hence, homomorphic encryption can enable services
to provide perfectly private inference.
I’ll try to say more about when and where this is useful later in this article.
First I wanted to give a more concrete sense
for how HEIR works in the context of these examples,
and outline (my view on) the project’s roadmap for the future.
I won’t do a deep dive on HEIR’s internals by any means,
since that would make the article too long.
Give me a shout if you want that, but there are plenty of docs
to read through at
and you can see a recent (fast-paced)
I gave at ASPLOS this year.
Table of Contents:
The repo behind the blog post
The blog post ends with a list of examples compiled with HEIR.
Those examples point to a GitHub repository2
that you can clone and run yourself.
The biggest hurdle is installing bazel,
and then bazel hermetically manages everything else.3
Some simple runtime comparisons
The simplest and fastest example to try is the
credit card fraud detector.
This is a simple three-layer feed-forward network with sigmoid activations,
trained on a Kaggle dataset.
The linear layers have dimensions 128, 64, and 2 (the last being the logits for the two classes, fraud and not-fraud).
You can run the basic example in one line:
This command will compile the (pre-trained, checked-in) cc_fraud model
to the Lattigo backend, and then
run it on a sample input. The command above outputs:
The central point here is that the evaluation of the model on encrypted inputs
took about 2 seconds on a single-threaded CPU.
Compare this to the same execution on cleartext inputs, noting that this is the
latency of a single inference, so it doesn’t benefit from amortization. (This
requires fetching and encoding the original dataset, which is explained in the
I’ll skip that part here).
Anyone who has heard of HE may have heard that it is slow, but I want to pause
here to compare this (single-threaded CPU, non-amortized!) runtime: 2 seconds
for HE inference vs 0.5 ms for cleartext. This is a 4,000x slowdown, and the
computation involves two matrix-vector products (where the matrix is not
private), with two evaluations of a sigmoid function.
There are many caveats to this demo worth briefly noting:
This model is small enough that the private information fits in a single
(CKKS) ciphertext. Inputs tensors with more than, say, 32k elements will
require multiple ciphertexts and naturally more model is small enough that bootstrapping (the slowest part of HE) is
not required.I did not include a simulated network overhead, nor did I include the
per-user one-time setup required to generate and upload the relevant key
material that makes HE server is doing a decent amount of (one-time, model-specific)
pre-computation in the “Running preprocessing” step.
The other examples in the repo are more complex, and hence have longer
latencies and worse overhead vs cleartext (and a memory requirement of 60-90
GiB). In particular:
network_anomaly, an ensemble of auto-encoders: 30 seconds for , a recommender model adapted specifically for HE: 5 minutes for , a 10-layer convolutional network: 20 minutes for inference.4
This sounds bad, but remember it’s single-threaded CPU execution. Our
colleagues working to integrate HEIR with GPUs have reported that the criteo
workload runs in ~500ms on a single GPU (similar to an H100).
Compare that to the criteo/cleartext:evaluate_cleartext demo which
runs in 10ms, and you’re down to a 50x slowdown (again, the baseline is
non-amortized CPU execution). That work wasn’t able to make it into the Google blog post,
but they are compiled by HEIR with some pending upstream PRs.
My point is that the execution times are continuing to improve, and
for some small problems they could be called reasonable if you squint.
And this does not even breach the topic of HE accelerated by FPGAs and ASICs,
which are even more promising performance-wise.
So instead of the Google corporate blog post showcasing raw performance,
it was meant to showcase the expressiveness of HEIR:
it can compile a lot of models,
and the performance on some of them is decent.
As far as showcasing features, the repo also shows how one might use HEIR
Target and compare multiple HE backends (the repo has Lattigo and OpenFHE as
examples, try bazel run -c opt //demos/cc_fraud/openfhe:evaluate_fhe.Explore per-layer timing: HEIR has the ability to insert debugging callbacks
into the compiled program, try bazel run -c opt //demos/cc_fraud/lattigo:evaluate_fhe_timing. In that command you’ll see the
first linear layer takes the majority of the runtime, ~1.2 seconds out of 2
seconds an incorrect inference: Similar to timing, the debug callback can
decrypt ciphertexts and inspect them for correctness or precision loss. Try
bazel run -c opt //demos/cc_fraud/lattigo:evaluate_fhe_debug and you’ll see
output that shows that by the end of the inference, the use of HE has caused
about 2 bits of precision loss compared to the cleartext model. This
precision loss can be tuned with compiler flags and trades against
performance. Unfortunately, navigating that trade-off requires HE expertise.
Getting a model into HEIR
The process of getting a pre-compiled model to be something that HEIR can process
is not yet automated. The main two constraints are:
You need to be able to convert the pre-compiled model to MLIR,
which is the intermediate representation that HEIR uses to represent programs. Many ML frameworks
such as PyTorch and JAX have tools to convert to need to manually annotate your program with HEIR-specific annotations
that say (a) what inputs to the inference are secret and (b) what are bounds
on the input ranges to each activation function.
The demos in the repository show how to do
for PyTorch, and I’m working with the maintainers of torch-mlir to add
that will enable me to automate this (given a validation set to use to estimate ranges).
That said, much of the early stages of the compiler pipeline (recognizing
activations, fusing linear layers, etc.) is based on how torch-mlir exports
models to MLIR, so if we want to add ONNX or JAX support (both have great MLIR
exporters), it will likely not work out of the box just yet.
Moreover, we don’t even have complete coverage of torch operators yet.
A lot of the details of how to support linear algebraic operators in HE are
both tricky and active research topics (for MLIR enthusiasts, we don’t support
in full generality).
Invoking the compiler
This is a bit of a tangent, but if you look at the build rules for the examples in the repo,
you’ll see some slightly messy calls to a macro that invokes the compiler.
What’s going on here is that HEIR’s interface is a lot more like
LLVM than clang. There are two binaries, heir-opt and heir-translate,
which handle running compiler passes and codegen, respectively,
matching LLVM’s opt and translate.
Those two binaries are wrapped into bazel rules,
which I published as
rules_heir,
and then further wrapped the rules in macros that
correctly stitch together the optimizer and codegen
binaries, and wrap the results into a cc_library, go_library, or
rust_library, as appropriate to the chosen backend.
One nice aspect of this is that rules_heir uses pinned binaries,
and can be pointed to a nightly or custom HEIR release. Still,
this is less of a clear user story than something like clang,
which has a consistent interface and limited exposure of the underlying
LLVM kitchen sink.
Our goal is to eventually make one (or multiple) clang-style products that have a
narrower scope (e.g., only supporting one frontend) and conceptually simpler
flags (like -O2). For frontends like JAX/PyTorch, this will probably be
wrapped in a Python library.
Speaking of Python libraries!
We do have a heir_py frontend library.
It’s completely unrelated to ML at this point.
It actually compiles a (very limited) subset of Python bytecode (using numba’s frontend)
to homomorphic encryption, and then runs HEIR, generates OpenFHE C++ code,
compiles and links that with clang (the slowest part!),
and then loads the machine code back into a Python module
which the user can call like a function.
To be clear, we haven’t used the Python frontend for anything particularly useful.
It still needs a lot of features to start being useful,
such as preserving numpy ops as MLIR linalg ops that HEIR understands.
Who wants HE?
Now I’d like to take a step back and pontificate a bit about who wants HE.
There has been a lot of discussion in the HE community about this,
and in particular the question has been raised: what is HE’s “killer app”?
Most academics will tell you the answer is “private LLMs.”
I will reserve my opinion on that topic for dinner parties,
but suffice it to say that two things are currently true:
First, most if not all applied HE work in academia is currently focused
on making encrypted LLM inference faster.
Second, the SOTA is still very far away from being practical.
In particular, latencies are measured in the units of seconds per token,
and all solutions I’m aware of require a round-trip to the client
to decrypt the output from one inference and prepare it for the next inference step.
And this is including hardware acceleration (e.g., 8 GPUs).
I’m also not an expert on LLM architectures, so I don’t have a strong
sense for the current barriers and how likely they are to
succumb to additional research efforts.
My focus at Google has instead been on a lower-hanging fruit:
what applications can take advantage of the HE capabilities we have today?
In that discussion, I have landed on the following soft heuristics,
which shed some light on the remaining challenges to make HE truly practical.
I will again restrict myself to private inference here.
Models from 2020
While the latency of HE has improved significantly,
models with billions of parameters are still out of reach
for practical applications. And in particular, this rules out
any transformers that are big enough to be useful.
That said, I have spoken with engineers and scientists at Google
who have informed me that, despite their efforts, their domain has NOT yet
seen any benefit from applying transformers.
As such, the core complexity of their model architectures
has not changed significantly since roughly 2020.
Convolutions are still king in many domains.
Or in other words, there are still useful tasks that
don’t need huge models. And so even if a large model could be used,
if privacy is important enough, a smaller one might be acceptable
to get fast HE.
Critical privacy for both parties
While I aspire to a world in which HE can be applied without a critical need for
privacy, we’re not there yet. So for HE to be useful,
the privacy of both parties must be critical.
I say “both parties” for the following reason.
The standard privacy protection one expects when discussing private inference
is sensitive user data.
If my email service can’t see the text of my emails,
I am spared from the risk of data leaks, insider threats,
exposure to warranted searches, etc.
The standard way to protect user privacy in these situations
is to do all the work locally on the user’s device.
In late 2024, Google Maps changed its location history feature
to be fully on-device,
at least partly in response to US police geofence warrants.
But for private inference in particular,
the trained model is also sensitive data of the service provider.
Shipping a model to a device, even one with a secure enclave,
risks exposure of the model weights to competitors.
For use cases like biometric authentication,
leaking the model also adds a security risk;
attackers can engineer attacks against the production model
in an unrestricted environment.
I’m not an expert on hardware-based security,
but some of my colleagues make a living breaking them,
and the way they talk about the topic
doesn’t inspire confidence in secure enclaves.
I have also been led to believe that there are use cases
where keeping a proprietary model secret exceeds the
importance of user privacy, because the user data in those
situations is not considered sensitive.
In these situations, “user privacy” is a bonus
on top of protecting IP.
That’s not as romantic as user privacy,
but hey, if it spurs further investment in HE,
I’ll put up with it to get closer to the ideal.
Not too many simultaneous users
As latency becomes less of a bottleneck for HE,
other bottlenecks naturally emerge.
The biggest one I see is key management.
The mathematics that makes HE work
requires special “evaluation keys.” These keys
are, quite literally, encryptions of the user’s
secret key (or something derived from the user’s secret key).
Ignoring for a moment the added assumption of
circular security,
what matters for practical HE systems is:
Evaluation keys are unique to each user of the scale with the program size in some sense (e.g., doing larger matmuls in HE
requires more evaluation keys than a smaller matmuls).The biggest examples from the HEIR demo repository need ~40 GiB of evaluation
keys. SOTA LLMs in HE need hundreds of GiBs of key material.
So if you had a million-user service with the 40-GiB-per-user use case,
you’d need 512x20T hard drives just to store the key material.
At ~500 USD per drive (just checked Amazon, so that’s a consumer price), you’re
asking for 250k USD extra storage costs. I’m sure the price of RAM needed to serve a
decent QPS is going to be worse.
And more, if you are using GPUs to accelerate the HE,
you have the added problem of getting this 40 GiB of key material
from disk to the GPU when a user makes a request.
My napkin math says this should take roughly
hundreds of milliseconds–the same order of magnitude as the
entire HE computation. And during the time that you’re filling up the GPU’s
memory with keys for one user, you can’t serve other user requests.
To be fair, a decent chunk of the RAM requirement consists of preprocessed
model weights, which are model-dependent and not user-specific.
And moreover, most of the key material required is to support bootstrapping,
so there may be good HE applications (like Apple’s)
that can use tricks to avoid needing to bootstrap.
Maybe in the best case we’d only need 1 GiB of key material.
So maybe you could fit enough keys for 50 users on a beefy datacenter GPU.
Still, you can see the systems bottlenecks here.
And I don’t think this is getting much better.
Newer HE schemes that promise better latency tend to require larger
key material, not smaller.
As such, until we (the HE community) finds a way to reduce the memory
requirements of HE by an order of magnitude,
I am only going to be looking for applications where there aren’t
too many simultaneous users.
The server shouldn’t need to see the output
A lot of people I’ve talked to who want to use homomorphic encryption
eventually realize that, actually,
the server wants to learn one bit of information about the user’s underlying cleartexts.
For example, say we want to authenticate a fingerprint without having to store the user’s biometrics.
HE would prevent the server from seeing the authentication result,
and so it can’t act on that.
One basic idea for dealing with this
is to send the encrypted authentication result
back to the user, have the user decrypt it,
and send the result back to the server.
The obvious downside to this is that
you can’t trust the client
to honestly report the authentication result!
To make this work, you would need the client
to generate a zero-knowledge proof that the
value they decrypted was actually the result of running the decryption routine
with the provided ciphertext and the user’s secret key.
It has to be zero-knowledge because
the server cannot learn anything about the user’s secret key.
While I haven’t done this myself,
I have heard it is both possible and there are still some challenges
in scaling it to HE schemes like CKKS.
And of course, it further eats away at a latency budget.
As such, near-term applications of HE are probably going to
have an easier time making it to production if the application
does not require the server to see the computation result.
So what applications meet these constraints?
Some ideas that seem to hold water:
Remote diagnostics. Say you run a factory and you have some expensive
machine you bought from a manufacturer. You want to let the manufacturer
diagnose the machine to recommend maintenance and identify problems,
but the details of what you’re using the machine to make (say, manufacturing
volume) or the settings of the machine itself would be somewhat sensitive
financial information if it were leaked to a competitor. However, diagnostic
reports are batch jobs that the server need not see the output of, there
aren’t millions of factories, diagnostics are
a premium service the manufacturer would want to charge for,
and they are probably not that complicated of models.
I have heard rumors that someone is actually using HE for this, but I have no
publicly verifiable proof. Our network intrusion detection demo is similar
in nature, where the contents of the packets are the sensitive data.5
B2B analysis: Similar to existing applications of private set intersection
at Google, business-to-business applications check a lot of boxes. Both businesses are privacy
needs, but there are only two users, so key material is not an obstacle. But
there is enough trust for the two parties to share the output, provided the
computation is producing some set of business metrics relevant to both parties.
And business analytics and metrics reporting are usually quite latency tolerant.
I recall talking to one HE practitioner who said, “if the client only needs
a report generated once a month, then who cares if the HE computation takes a
week to run?”
Specialized biometrics situations: I think biometric applications like face
and fingerprint recognition are within the capabilities of current HE
techniques, with the exception of the key material problem. But there are
specialized situations in which extra latency should be tolerated, such as
in-person account creation or account recovery procedures
(think of a new employee on-boarding day, or losing your security badge).
These events are infrequent enough that the system wouldn’t be encumbered by high
QPS needs. And I think it is generally a good idea that employers do not have
databases of employee biometrics.
Some notes on the HEIR roadmap
I’ll close with a few notes on where HEIR is going next.
Crypto in the compiler
You may have noticed all the HEIR backends are currently external library APIs
in a high level language. This is mainly because, when we started HEIR,
hardware acceleration work was still in its infancy.
Moreover, people building accelerators (and GPU libraries) needed an easy
way to write larger applications that use their libraries for benchmarking
purposes. These often mirrored some existing library API (like OpenFHE),
and then injected their own code at some level, such as scheduling a
key switching op on a GPU.
So we decided HEIR should meet them in the middle.
Start by targeting these popular library APIs,
and get the frontend of the compiler pipeline working well.
At the same time, we have been slowly building out a true “backend”
of the compiler, which involves implementing the HE cryptography directly
as compiler passes. With that in place, and with our multi-layer design
enabled by MLIR,
we can compile a program down to a library API (high-level HE scheme
operations), or to a lower-level modular polynomial arithmetic
which some accelerators use as their starting point,
or to an even lower-level vectorized modular arithmetic level
(think of a GPU or TPU for 64-bit integer math modulo custom primes, instead of bfloat16)
which some accelerators use as their starting point,
or even further down to LLVM, and then to x86 or wasm or whatever.
This will be critical for multiple of our hardware partners,
including Niobium, Cornami, and Optalysys,
all of whose hardware we are planning to target with HEIR.
This path is making steady progress, but I expect it to pick up
given that we have both a lot more GPU work maturing, and that I personally should
have time in the next 6 months to really dig into that work.
That will in turn open a lot of potential for optimizations
that involve scheduling, kernel fusion, etc.
GPU and TPU support
We have a bunch of parallel efforts at integrating TPU and GPU backends
into HEIR (still as library APIs for now). This includes
the CHEDDAR library,
Belfort’s GPU library (I’m not sure if their project codename is public),
and CROSS/jaxite.
We also have heard interest from a variety of other
groups working on GPU
acceleration.
In my view, GPU/TPU will be crucial to getting HE in production
before custom ASICs can be scaled up in datacenters.
New schemes
Researchers are coming out with new schemes every few years,
and HEIR is meant to support all of them. Some that are on our
radar include
Gentry-Lee which is optimized
for matrix multiplication and has a lot of potential for GPU/TPU which streamlines and
simplifies a lot of the underlying polynomial math of CKKS, and also
makes switching between schemes much simpler. This has already had some
-Zheng which uses a fancy encoding
to enable both arithmetic and logical style operations in one scheme.
Benchmarking
One of the big selling points of HEIR is benchmarking.
In particular, I want to be able to benchmark all the different
hardware solutions, HE schemes, and optimization techniques
so that I can be informed enough to decide what special hardware
Google should integrate into its datacenters (when the time comes).
As part of that my colleague
Shruthi Gorantala has been hard at work
aligning the industry around a benchmarking solution in the form of
.
We want to add features to HEIR so that it can generate code in the right
structure to be submitted to the benchmarking repository. This would make
it very easy for people working on HEIR (particularly researchers)
to test and showcase the benefits of their optimizations.
A huge part of the draw of HEIR is that researchers can implement their
research ideas directly in HEIR, or at least use HEIR as a frontend.
So a big part of my job is to help train graduate students on working with HEIR,
reviewing their code, etc.
Though HEIR still doesn’t have a proper publication yet, it has been racking up
and, to my count, four published papers were based on research done directly in HEIR.
If this interests you, or if you want to get some experience in
compilers/MLIR or cryptography, HEIR is fully open source,
and we have both a weekly office hours and a
(recorded) monthly meeting
where we review work that happened in the past month and discuss upcoming design
work. Everyone is welcome, but we just ask that you show up to one of these
in-person meetings (or schedule a private meeting with a maintainer)
before submitting a PR,
which is part of our policy to reduce AI
.png)
