Apache NiFi Expression Language and the use cases

One of the greatest features NiFi has given us, is the expression language. This feature allows us to program our dataflow while dynamically changing the behavior towards the FlowFile, according to the FlowFile we transfer.

It always starts with a dollar sign and an opening curly brace(“${“) and ends with a closing curly brace(“}”). The first variable would be either an attribute from the FlowFile(first priority) or from the Variable Registry. Then we can add a colon and a function. Example: ${variable:function()} .

NiFi allows us to use a variety of functions that allow us to be extremely dynamic with our flow. It can produce booleans, strings, and even a list. My most used function is now() which produces the current date. Then, usually I change it to number using toNumber() or format it according to the format I need using format().

With updateAttribute, we can even update the state manager(using Store State property) and query the state manager while updating it(using getStateValue()).

When a variable doesn’t exist, NiFi would just generate an empty string, and because of that, people might think that once an expression language is valid, it cannot fail. However, this is not true. That’s why the fact that UpdateAttribute has only one relationship(by default) which is named success, is a bit illusive.

Let’s test it: Let’s set up a GenerateFlowFile with an invalid JSON as an attribute.

notice the missing curly brace at the end

Now, set up an UpdateAttribute that queries the attribute from before using the function jsonPath(), which allows us to query a JSON using JSON Path.

Now, run the flow:

And, Voilà! UpdateAttribute, against all odds and with a totally valid expression language, has failed! Well, obviously it can’t parse an invalid JSON. So yes, even a valid expression language can fail.

But it doesn’t mean the expression language is any less powerful. In my company, we are working on a project, some sort of a rule engine. This engine receives JSON files, like this one:

We need to apply dynamic rules upon them to detect if we receive an invalid data. For example, type “8”, is a device that should not reach 30 degrees. So if we receive “temp” that is over 30 and the timestamp is within the last hour, then something is wrong.
Our rules must be dynamic, as we might add additional rules in time. So we set up a JSON file of rules that we can change at any time. But now the engine’s logic can be complicated, as we receive a variety of data.

So, I needed to find a way(preferably the easiest) to make up such engine. Then I remembered, NiFi has the expression language that can do so much! So I’ve added the dependency and added the relevant code!

the dependency
The code. I’ve used StandardPropertyValue since we have rules that apply to all and I want “true” to be evaluated too.

And now, I can use NiFi’s expression language in our rules:

In conclusion, NiFi’s expression language is very advanced, and can be even used outside NiFi(although I don’t think the developers thought it would be). I would even add it as an apache-common. You can read more about the NiFi Expression Language in the official guide and I recommend reading it.

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

QuickTip : Why is my disk usage steadily growing when using docker?

Vulnhub DriftingBlues:7 write-up

How to setup Phabricator on AWS (Part 2 — Launching Phabricator)

Two Years Later, I Finally Saw Useful Software Testing

What is iPerf?

The OLE DB Provider “Microsoft.ACE.OLEDB.12.0” for Linked Server “(null)”

Using the Symfony Serializer with Doctrine relations — Part 1

Run your ownCloud on any server with Containers and Cloud 66

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ben Yaakobi

Ben Yaakobi

More from Medium

How much memory does a Java server actually need? (Part 3/3)

Weak References: A weapon against Out Of Memory Errors

Technical Series: Introduction to Optimizing Performance

What is containerization and what role does Docker play in it?