Skip to main content

Collections

NQE provides three types for working with collections of values.

TypeDescription
BagAn unordered collection of elements where elements may appear any number of times.
ListLike Bag, but the elements are arranged in a specific order.
SetLike Bag, but each element can appear only once. In other words, an unordered collections of elements where multiplicity does not matter.

While Bag and List are generic datatypes that support any type of element, the Set datatype is currently focused on representing sets of IP addresses. Bags and Lists support iteration, but Sets do not. This remainder of this guide focuses only on Bags and Lists. See the guide for Sets for more information on Sets.

The primary difference between Bags and Lists is that Lists are ordered collections, whereas bags are unordered. Ordered collections have special operations that make use of the fact that the collection is a sequence of values. Examples include list indexing and limit.

The following query retrieves all devices, sorts them by name, and limits the results to the first 10:

foreach device in network.devices
select { Name: device.name, Platform: device.platform }
order by Name ascending
limit 10
Important Language Changes

NQE 25.11 introduced stricter typing for Bags and Lists that may affect existing queries. If you have queries written before 25.11, please review the Breaking Changes and Migration section.

Table of Contents​

Breaking Changes and Migration​

Changes in NQE 25.11​

NQE 25.11 introduced stricter typing rules for Bags and Lists that may affect existing queries. In particular, some collection-valued expressions are now typed as Bags rather than Lists. This section outlines the key changes and how to migrate your code.

Bag/List Comparisons Generate Warnings​

Comparing a Bag with a List using equality (==, !=) or relational operators (<, <=, >, >=) is allowed but discouraged ─since List comparisons care about the positional order of elements, while Bag comparisons are order-insensitive. Such comparisons will generate warnings, and you should fix them with one of the suggested actions.

When you compare a Bag with a List, the List is implicitly converted to a Bag (losing its order), and then Bag comparison semantics are used. This can lead to unexpected results if you intended to compare order-sensitive Lists.

Example of changed behavior:

Before 25.11: This worked without any issues.

deviceNames = foreach device in network.devices select device.name;
{[
    isEqual: deviceNames == ["device1", "device2"]
]}

Here, isEqual is true precisely when there are two devices, the very first is named "device1" and the very last is named "device2".

After 25.11: Square brackets “[…]” are typed as Lists. As such, comparing a List with a Bag uses Bag comparison semantics (order-insensitive), which could give you unexpected results. As such, you will get a warning.

deviceNames = foreach device in network.devices select device.name; // This is now a Bag of strings
{[
    isEqual: deviceNames == ["device1", "device2"]
]}

Here, isEqual is true precisely when there are two devices, one is named "device1" and the other is named "device2", in no specified order.

Migration strategy: Make both operands the same type!

Option 1: If order matters then explicitly use order by to get a List.

// Use “order by” clause to sort the devices alphabetically by name
deviceNames = foreach device in network.devices
              let name = device.name
              select name
              order by name;

{[
    isEqual: deviceNames == ["device1", "device2"]
]}

Option 2: If order does not matter then use bag to get a Bag.

deviceNames = foreach device in network.devices select device.name; // This is a Bag of strings

// Ensure the comparison is done against two bags.
{[
    isEqual: deviceNames == bag(["device1", "device2"])
]}

Creating Collections​

There are several ways to construct Bags and Lists in NQE.

Literals​

The simplest way to create a list collection is by writing a list literal, a comma-separated sequence of expressions within brackets:

numbers  =  [1, 2, 3, 4, 5];
names    =  ["router1", "router2", "router3"];
mixed    =  [[10, 20], [30, 40]];  // list of lists

The Bag function​

We can wrap a list literal with the bag method to create a bag collection.

integers  =  bag([1, 2, 3, 4, 5]);  // a bag of integers

Since bags ignore order, the following expression is true:

bag([1, 2, 1])  ==  bag([2, 1, 1])

Comprehensions​

Collections can be created using foreach ... select ... expressions, also known as comprehensions.

A comprehension evaluates to a Bag if the source collection is a Bag (and there is no order by clause).

// When iterating over a Bag (like network.devices), the result is a Bag
deviceNames  =  foreach device in network.devices select device.name;  // Bag<String>

A comprehension evaluates to a List if the source collection is a List (or if there is an order by clause).

// When iterating over a List, the result is a List
numbers  =  [1, 2, 3, 4];                       // List<Integer>
doubled  =  foreach n in numbers select n * 2;  // List<Integer>: [2, 4, 6, 8]

Concatenation​

Collections can be combined using the + operator, which concatenates two collections: It preserves order if both collections are lists, and otherwise produces a bag.

list1     =  [1, 2, 3];
list2     =  [4, 5, 6];
combined  =  list1 + list2;  // == [1, 2, 3, 4, 5, 6]

Combining Bags and Lists results in a bag:

myList    =  [1, 2, 3];
myBag     =  bag([4, 5, 6]);
combined  =  myList + myBag;  // == bag([1, 2, 3, 4, 5, 6])
                              // == bag([2, 4, 6, 5, 3, 1])

Difference​

The - operator removes all elements from the first collection that appear in the second one. The resulting difference is an ordered collection exactly when the first argument is ordered.

Removing from a bag results in a bag:

all          =  bag([1, 2, 3, 4, 5, 4]);
exclude      =  [2, 4];
difference   =  all - exclude;  // == bag([1, 3, 5])
                                // == bag([5, 1, 3])

Removing from a list results in a list:

all         =  [1, 2, 3, 4, 5, 4];
exclude     =  bag([2, 4]);
difference  =  all - exclude;  // == [1, 3, 5]
note

Notice that the item 4 is duplicated in all, however the resulting difference removes all occurances.

Builtins That Return Collections​

Many builtin functions return collections as their result. For example:

  • fromTo(start, ebd): Returns a list of numbers from start to end (inclusive).
  • regexMatches(text, regex): Returns a list of all regex matches found in the text
  • splitJsonObjects(json): Returns a list of JSON object strings
  • patternMatches(config, pattern): Returns a bag of pattern matches from configuration
  • blockMatches(config, pattern): Returns a bag of block matches from configuration

See the standard library reference for more functions that return collections.

Converting Between Bags and Lists​

bag Function​

Convert a List to a Bag using the bag(list) function:

orderedDevices = [/* ... */];     // List
deviceSet = bag(orderedDevices);  // Bag

This is useful to make explicit the implicit downcasts when using ==, !=, <, <=, =>, or > with a Bag and a List, which produce a type check warning and will eventually produce a type check error. See Comparing Bags and Lists for more details.

Lists are subtypes of Bags​

Since a list is just a bag with a specified order, a list can be used anywhere a bag is expected ---we just "forget" the ordering of the list to obtain a bag.

If-Then-Else: When branches return different types:

aList      =  [1, 2, 3];                          // List<Integer>
aBag       =  bag([3, 4, 5]);                     // Bag<Integer>
condition  =  false;
combined   =  if condition then aList else aBag;  // Bag<Integer>

User-Defined Functions: When a function expects a Bag parameter but receives a List:

processDevices(devices: Bag<Device>) : Integer = length(devices);

[{ useCase1: processDevices(["Device 1", "Device 2"])
 , useCase2: processDevices(bag(["D 10", "D 20"]))
 }]

Notice that useCase1 works since we automatically get to use processDevices as if it were typed processDevices(devices: List<Device>) : Integer as well ---since List is a subtype of Bag.

List Indexing​

Lists support zero-based indexing to access individual elements. Use square brackets with an integer index to retrieve an element at a specific position.

numbers  =  [10, 20, 30, 40, 50];
first    =  numbers[0];    // 10
second   =  numbers[1];    // 20
last     =  numbers[4];    // 50

When an index is out of bounds (negative or greater than or equal to the list length), a runtime error is emitted.

numbers    =  [10, 20, 30];
tooLarge   =  numbers[10];   // 🚫 Error: Index too large
negative   =  numbers[-1];   // 🚫 Error: Index must be non-negative

Notes:

  • List indexing only works with Lists, not Bags (bags are unordered and don't support indexing)
  • Indices must be integer values
  • Attempting to index a bag results in a type error
  • Out-of-bounds access results in a runtime error

Ordering​

One of the most powerful features of Lists is the ability to specify and maintian order.

The order by Clause​

The order by clause sorts a comprehension by one or more expressions, resulting in an ordered collection.

1. foreach device in network.devices
2. select { Name: device.name, Platform: device.platform }
3. order by Name asc

Notice that line (3) orders all records specified in line (2) by the Name field in ascending order.

To order by names in descending order, we can use the desc keyword:

1. foreach device in network.devices
2. select { Name: device.name, Platform: device.platform }
3. order by Name desc

Instead of asc and desc, for readability you can use ascending and descending to specify sorting direction.

1. foreach device in network.devices
2. select { Name: device.name, Platform: device.platform }
3. order by Name ascending

You can sort by multiple columns: Line (3) below orders by the Platform field of each record, specified in line (2), in descending order and then orders by the Name field of each record in ascending order.

1. foreach device in network.devices
2. select { Name: device.name, Platform: device.platform }
3. order by Platform desc, Name asc

Instead of asc and desc, for readability you can use ascending and descending to specify sorting direction.

1. foreach device in network.devices
2. select { Name: device.name, Platform: device.platform }
3. order by Platform descending, Name ascending

Finally, you can even sort by local let variables:

/** 😲 Results are ordered by platform ascending,
  *    then by model descending, then by name;
  *    however, the results only show the name and platform. */
foreach device in network.devices
let Platform = device.platform
// 🔽️ This is used for sorting only & not shown in the rows.
let Model = Platform.model
select {
  Name: device.name,
  Platform
}
order by Platform asc, Model desc, Name ascending

Important: The order by clause requires each ordering expression to be a name that refers to either:

  • A field of the record returned by the select clause, or
  • A variable defined in a let clause

You cannot use arbitrary expressions directly in order by. If you need to sort by a computed value, bind it to a name using let first.

bad    =  foreach device in network.devices
          select device
          order by device.name; // 🚫ERROR: “order by” clause expects an identifier, not an expression

works  =  foreach device in network.devices
          let name = device.name
          select device
          order by name; // 🟢 Works

The orderBy Function​

To sort collections directly, use the orderBy(collection, keySelector) function:

getDeviceName(device) = device.name;

devicesSortedByName = orderBy(network.devices, getDeviceName);

For descending order, negate numeric keys or use the comprehension syntax.

The order Function​

To sort a collection by its natural order without specifying a key selector, use the order(collection) function:

devices = order(network.devices);

The order function sorts elements using their natural ordering - the same ordering used by comparison operators like < and >. See Comparisons for details on how different types are ordered.

This is equivalent to the following comprehension:

devices = foreach device in network.devices
          select device
          order by device ascending;

The order function always returns a List in ascending order. For descending order, use the comprehension syntax with order by ... descending.

Limiting Results​

Basic Use​

The limit clause restricts the number of elements returned from a list:

foreach device in network.devices
select { Name: device.name }
order by Name ascending
limit 10  // Only first 10 devices

Without An order by Clause​

While limit is commonly used with order by to get the "top N" results, it can be used without order by as long as the comprehension produces a list. For example, iterating over an already-ordered list will allow limit to work without requiring order by. For instance, if you already have your devices ordered, then you can do:

devicesList = ["device1", "device2", "device3"];

foreach device in devicesList
select { device }
limit 2  // No “order by” clause needed since “devicesList” is a list

Limiting by an expression​

The limit value can be any expression that evaluates to a number, not just a numeric literal. For example, the following query gets half of the possible devices.

foreach device in network.devices
select { Name: device.name }
order by Name ascending
limit length(network.devices) / 2

Note: The order by and limit clauses can appear in nested queries, not just at the outermost level. This allows you to sort and limit intermediate results within a larger query.

The limit function​

The limit function can also be applied directly to collections:

getDeviceName(device)   =  device.name;
devicesSortedByName     =  orderBy(network.devices, getDeviceName);
firstFiveDevicesByName  =  limit(devicesSortedByName, 5);

Operations​

Membership Testing​

Test whether an element is in a collection using in or not in:

[{ present: "Jasim" in ["Jane", "Jasim", "Jerry"] // = true
 , notPresent: 46 not in bag([3, 7, 365])         // = true
}]

This can be used in comprehensions to filter out results:

invalidDevices = ["Device1", "Device5", "Device9"];

foreach device in network.devices
let Name = device.name
where Name not in invalidDevices
select { Name }

Comparison Operators​

Collections can be compared using <, <=, >, >= operators.

  • Lists are compared lexicographically (element by element).
  • Bags are compared by sorting them (using the natural order) then doing list comparisons.
[{ true1: [1, 2, 3] < [1, 2, 4]           // = true since 3 < 4 at first differing position
 , true2: [1, 2]    < [1, 2, 4]           // = true since first list is a prefix of the second list
 , true3: bag([2, 1, 1]) < bag([1, 3, 1]) // = true since [1, 1, 2] < [1, 1, 3] is true
}]

Equality​

Lists are equal if and only if they have the same elements in the same order:

[1, 2, 3]  ==  [1, 2, 3]     // = true
[1, 2, 3]  ==  [3, 2, 1]     // = false; different order
[1, 2, 3]  ==  [1, 2, 3, 4]  // = false; different length

Bag equality is order-independent but cares about multiplicity:

bag([1, 2, 3])  ==  bag([3, 2, 1])  // = true; same elements, order ignored
bag([1, 2])     ==  bag([2, 1])     // = true; same elements, order ignored
bag([1, 2])     ==  bag([2, 1, 1])  // = false; “1” occurs more times in the right than in the left

Comparing Bags and Lists​

Comparing a Bag with a List using equality or relational operators is allowed but generates a warning. When comparing a Bag with a List, the List is implicitly converted to a Bag (losing order information), and Bag comparison semantics are applied.

myList  =  [1, 2, 3];
myBag   =  bag([3, 2, 1]);

myList == myBag  // ⚠️ Warning: comparing List with Bag
                 // Result: true (order is ignored, both become bags for comparison)

To avoid warnings and make your intent clear, convert both operands to the same type:

// If order matters
myList == order(myBag)  // No warning; false since [1, 2, 3] != [3, 2, 1]

// If order doesn't matter
bag(myList) == myBag    // No warning; true

For more details about language changes that may affect existing queries, see Breaking Changes and Migration.

Extracting Single Elements​

Use the(collection) to extract a single element from a collection. This function returns:

  • The single element if the collection contains exactly one element
  • null if the collection is empty or contains more than one element
// The device named "core-router-1", if present; else null
coreRouter1Device = the(
  foreach device in network.devices
  where device.name == "core-router-1"
  select device
);

This is particularly useful when you expect a unique result but want to handle the case gracefully when the assumption doesn't hold.

Deduplication​

Remove duplicate elements from a collection using distinct. For lists, this preserves order: The first occurance of an element is kept and all other subsequent occurances are dropped.

list        =  [1, 2, 3, 1, 4, 2];
uniqueList  =  distinct(list);  // [1, 2, 3, 4]

bag        =  bag([1, 2, 3, 1, 4, 2]);
uniqueBag  =  distinct(bag); // == bag([1, 2, 3, 4])
                             // == bag([2, 4, 3, 1])

List-Specific Data Model Fields​

Several data model fields return Lists instead of Bags to reflect their inherent ordering. Examples include:

  • Device.aclEntries and Device.natEntries
  • IfaceAcls.inboundAclNames and IfaceAcls.outboundAclNames
  • NetworkAcl.ingressRules and NetworkAcl.egressRules
  • SecurityGroup.ingressRules and SecurityGroup.egressRules
  • LoadBalancer.loadBalancerRules and LoadBalancer.outboundRules

See Also​

Guides​

Functions​

The following operations are available on collections.

  • fromTo(start, end) Creates a range of numbers [start, start + 1, start + 2, ..., end].

  • bag(list): Converts a list to a bag

  • length(collection): Gets the number of elements in a collection

  • limit(list, n): Gets the first n elements from a list

  • item in collection: Checks that item is an element of collection. For example, "b" in ["a", "b"] returns true.

  • item not in collection: Checks that item is not an element of collection.

  • distinct(collection): Removes duplicates from a collection.

  • orderBy(collection, keySelector): Sorts a collection by applying the key selector function to each element. The result is a list.

    • This can be written as a comprehension 
         orderBy(collection, selector)
      =
         foreach item in collection
         let key = selector(item)
         select item order by key ascending
    • Often, we want to order a collection by the natural order on elements; that is, the same order used for comparing items with relational expressions such as less-than (<) and greater-than (>). See Section comparisons for more detail on the order of values. This is done by order(collection) ---no selector function required.

  • max(collection): Gets the maximum element in a collection, or null if the collection is empty.

    • Note: max(collection) is the last element of order(collection).

  • min(collection): Gets the minimum element, or null if the collection is empty.

    • Note: min(collection) = order(collection)[0]

  • maxBy(collection, keySelector): Gets the element with the maximum key value, or null if the collection is empty.

    • This gets the greatest item in the collection, where items are ordered by the value returned by applying keySelector to each item.
    • Note: maxBy(collection, keySelector) is the last element of orderBy(collection, keySelector).

  • minBy(collection, keySelector): Gets the element with minimum key value, or null if the collection is empty.

    • This gets the least item in the collection, where items are ordered by the value returned by applying keySelector to each item.
    • Note: minBy(collection, keySelector) = orderBy(collection, keySelector)[0].

  • the(collection): Extracts the single element from a collection, if it has a unique element; else null.

  • sum(collection): Sums the numeric elements in a given collection.

  • join(delimiter, collection): Concatenates together the string elements in collection using delimiter between each element.

  • any(collection): Checks if any boolean element is true

  • all(collection): Checks if all boolean elements are true