Apache NiFi Expression Language and the use cases

What is this expression language? Is it totally exception proof when the syntax is valid? And how did we use it in our own project?

One of the greatest features NiFi has given us, is the expression language. This feature allows us to program our dataflow while dynamically changing the behavior towards the FlowFile, according to the FlowFile we transfer.

It always starts with a dollar sign and an opening curly brace(“${“) and ends with a closing curly brace(“}”). The first variable would be either an attribute from the FlowFile(first priority) or from the Variable Registry. Then we can add a colon and a function. Example: ${variable:function()} .

NiFi allows us to use a variety of functions that allow us to be extremely dynamic with our flow. It can produce booleans, strings, and even a list. My most used function is now() which produces the current date. Then, usually I change it to number using toNumber() or format it according to the format I need using format().

With updateAttribute, we can even update the state manager(using Store State property) and query the state manager while updating it(using getStateValue()).

When a variable doesn’t exist, NiFi would just generate an empty string, and because of that, people might think that once an expression language is valid, it cannot fail. However, this is not true. That’s why the fact that UpdateAttribute has only one relationship(by default) which is named success, is a bit illusive.

Let’s test it: Let’s set up a GenerateFlowFile with an invalid JSON as an attribute.

notice the missing curly brace at the end

Now, set up an UpdateAttribute that queries the attribute from before using the function jsonPath(), which allows us to query a JSON using JSON Path.

Now, run the flow:

And, Voilà! UpdateAttribute, against all odds and with a totally valid expression language, has failed! Well, obviously it can’t parse an invalid JSON. So yes, even a valid expression language can fail.

But it doesn’t mean the expression language is any less powerful. In my company, we are working on a project, some sort of a rule engine. This engine receives JSON files, like this one:

We need to apply dynamic rules upon them to detect if we receive an invalid data. For example, type “8”, is a device that should not reach 30 degrees. So if we receive “temp” that is over 30 and the timestamp is within the last hour, then something is wrong.
Our rules must be dynamic, as we might add additional rules in time. So we set up a JSON file of rules that we can change at any time. But now the engine’s logic can be complicated, as we receive a variety of data.

So, I needed to find a way(preferably the easiest) to make up such engine. Then I remembered, NiFi has the expression language that can do so much! So I’ve added the dependency and added the relevant code!

the dependency

The code. I’ve used StandardPropertyValue since we have rules that apply to all and I want “true” to be evaluated too.

And now, I can use NiFi’s expression language in our rules:

In conclusion, NiFi’s expression language is very advanced, and can be even used outside NiFi(although I don’t think the developers thought it would be). I would even add it as an apache-common. You can read more about the NiFi Expression Language in the official guide and I recommend reading it.