GR:Gravity/Xref/Queries

Queries: Searching in sources

Running a Simple Xref query

From the Gravity or TD/OMS application you can run a simple Xref query be selecting the Xref tool button '</>' in the top toolbar. The toolbar button opens the Simple Xref Query dialog. Before you hit this button or any other Xref menu or button make sure the Gravity Xref server has been configured in the Xref preferences (this step is not necessary when using the Gravity or TD/OMS client).

In the Xref Search field, you can enter the text which you want to cross-reference (using all the configured locations). For example, the search text ‘cdxferxc’s as entered below resulted in 7 hits.

The above query will run a full-text search of 'cdxferc' across all available locations.

With the exact option enabled a full-text search is enhanced to overcome limitations imposed by certain special characters in the text being searched. The find with the exact match will attempt to find the text as is including any special characters and preservation of upper and lower case characters.

NOTE. A full-text search is always case insensitive even if you enclose the text in double-quotes. To get results based on a case sensitive text then use the exact match option.

Xref query results view

Below is the output of the example simple Xref query:

A hit is an Xref entry in which the searched text was located 1 or more times. Each entry will show the Xref location where the source is located and the path to the source file. Expanding the hit will reveal the source line(s) in which the text was located.

if you double click on a source line-entry then that line will be shown as the first line in the workspace editor or web page (depending on the preference setting 'Match Xref source file to workspace file').

Note. If the Xref preference 'Match Xref file to workspace file' is checked then not the Xref location but the workspace project name will be shown. This will only happen if the Xfer runtime was able to map the Xref file to a local workspace file.

Depending on the hit file source type and preference settings additional menu options will be displayed when right-clicking on the hit file entry, for example:

Drill down from an Xref hit

The drill-down menu options available on an Xref hit will run a search using the base file name (without the extension) of the selected hit (source file). This is useful for those cases where the source file name can be used to drill down to the next level of search results. This can be done when the Xref source file hit is for example a Java source file or RPG program where the base source file name can be used as the input for an Xref query. By default Find all and Find... will be made available and for known Xref source types the options Find references and Find definitions are added to the available menu options.

Find references

For known Xref source types (see Xref Language Type support) a search will be done to find source files containing a reference to the search text.

Find definitions

For known Xref source types (see Xref Language Type support) a search will be done to find source files containing a definition of the search text.

Find all

Use the file source name as the primary criteria to do a (simple) search over all Xref locations.

Find...

Use the advanced Xref query wizard to fine-tune a search using the source file name as the primary search criteria.

View in editor

When the source file of the Xref hit belongs to a library (lib) or TD/OMS (tdoms) Xref location then the View source option will be made available. The source shown will be the source as it is known within TD/OMS, either the library source member or the IFS source file.

Export to CSV

With the export to CSV (comma separated values) 1 or more selected hits will be exported to a CSV file format. Any selection can be made of entries shown in the view, for example:

The file name used for the export is derived from the xref query string prepended with Xref. The query string used in the file name will be limited to 60 characters (due to the fact that a query string can be very long).

The CSV layout is Type, Name, Location, LineNo(s) and LineData. The type will refer to the selected entry if it is a file name entry then File will be set else Line for a line entry. Depending on the type the LineNo(s) will contain the total number of hits for a File else the line number if a Line. For a type Line the LineData will be set with the line data as shown in the view, for a type File the LineData will be empty. The Location refers to the location of the file on the Xref server, the first segment of the Location is the Xref location name (as defined in the Xref UI).

Example output of above selection:

Enhancing a simple search query

The simple Xref query runs a full-text search using the provided word or words.

The syntax that can be used in a search is based on the Lucene Query Language where a combination of words and operators are called a clause. A clause can be a simple term (word) or a specific search for terms in a field. A query can be enhanced with wild cards and operators to include advanced Xref searching capabilities. For a more thorough explanation of the Lucene Query Language see the section Xref Search Capabilities.

An alternative for the simple Xref query dialog is the Advanced query dialog which has support for all the Xref query fields and allows a search based on individually set fields, see Advanced Query Dialog. From the advanced dialog, the following query fields can be set:

full-text search
search for a definition
search for a symbol
include a file path(s) in the search
include a source type in the search

From the advanced dialog, you can also set the locations to be included in a search, if none is set then all locations are used in the search.

The ability to use the different terms in a search is highly dependent upon the scanned and indexed source files. If a source file is not of a supported language type then only a full-text search will be possible. See the below Language Type support for the list of source language types that will be analyzed with respect to definitions and references.

Language Type support

You can limit the search to a specific language type by providing the 'type' field. For example by selecting the type 'java', the query will be limited to searching in the files with the extension 'java'. The following languages have type definition:

c = C
clojure = Clojure
csharp = C#
cxx = C++
erlang = Erlang
fortran = Fortran
golang = Golang
haskell = Haskell
java = Java
javaclass = Java class
javascript = JavaScript
json = Json
kotlin = Kotlin
lisp = Lisp
lua = Lua
pascal = Pascal
perl = Perl
php = PHP
plain = Plain Text
plsql = PL/SQL
python = Python
rust = Rust
scala = Scala
sh = Shell script
sql = SQL
tcl = Tcl
troff = Troff
vb = Visual Basic
xml = XML

NOTE. If a type is not in the list of supported languages then the file types to be included in a search can be set using a file extension in the path field. Be aware that when using the term 'type' in combination with the term 'path' that the last will take precedence.

Advanced Query Dialog

The advanced query dialog extends the simple Xref query dialog by including the search terms as dialog fields in the Query Definition section. The dialog also has a Query Filter section that includes a type selection drop-down box and of which the 'Locations' field is a button allowing you to select 1 or more groups and or locations that should participate in the query. From the 'Locations' selection box a group is recognized by its surrounding angular brackets, for example <Xref_Group_1>. The dialog also has support for query history, which allows the user to retrieve a previously run query and rerun it again or to use the query as the base for a new query.

Depending on the type of search required you can fill-in a clause or simple search text in the fields Full Search, Definition, and Reference. Optionally from the Query Filter section a language type or path can be applied to limit the search to a specific file type, location(s), or file path(s).

With the button 'Show Query' the query search string will be displayed, this is also the string that will be passed on to Xref the query engine.

In the Query History section, you can recall a previously run query string. With the load button, the string will be disassembled and the values loaded into the dialog. A maximum of 50 queries can be stored in the query history, once the limit of 50 has been reached the oldest query will be dropped to make room for a new query. The query history can be cleared by pressing the 'Clear History' button, this is a permanent action, after you hit this button the deleted history can't be recovered.

NOTE. Similar to the Simple Query Dialog, clauses can also be applied in the Advanced Query Dialog but you do not have to include the terms (full:, defs:, refs:, path: or type:) as part of the clause, the terms will automatically be set in the final Xref query string.

Advanced Query Syntax

Lucene is the Java-based indexing and search technology used by Xref. Xref search capabilities are an extension of the Lucene query language which provides a rich set of search facilities to extract the most from the indexed Xref locations.

Terms

A query is broken up into terms and operators. There are three types of terms: Single Terms, Phrases, and Subqueries.

A Single Term is a single word such as "hello" or "sir".

A Phrase is a group of words surrounded by double quotes such as "hello sir".

A Sub query is a query surrounded by parentheses such as "(hello sir)".

Multiple terms can be combined together with boolean operators to form complex queries (see below).

Fields

Lucene supports fields of data. When performing a search you can either specify a field, or use the default full (text) field. The field names depend on indexed data, for example '''refs''' for references or '''defs''' for definitions. Whether a project has def or ref fields indexed, depends on the availability of an Xref analyzer for the source file being indexed. See the Language type support for a list of included analyzers ([[#Language Type support|Language Type support]]). The following fields are available in Xref:

full, full text search
defs, search for a definition
refs, search for a reference or symbol
path, query including a path part
type, query including a source type

These fields can only be used in the simple Xref query dialog, The advanced query dialog already separates these fields into separate dialog parts which makes it more easy to enter certain query data such as source types and target source locations.

You can search specific fields by typing the field name followed by a colon ":" followed by the term you are looking for.

refs:"userTable" AND full:userName

Note: The field is only valid for the term, phrase or sub-query that it directly precedes, so the query

refs:”userTable” user name

Will only find "userTable" in the def field. It will find "user" and "name" in the default full field.

Term modifiers

A query term can be modified to provide a wide range of searching options.

Boolean Operators

Boolean operators allow terms to be combined through logic operators. Lucene supports AND, "+", OR, NOT and "-" as Boolean operators. The boolean operators NEED to be ALL CAPS.

AND, OR, and NOT operators and "+", "-" defines two different styles to construct boolean queries.

If the AND/OR/NOT style is used, then an AND or OR operator must be present between all query terms. Each term may also be preceded by NOT operator. The AND operator has higher precedence than the OR operator.

AND

The AND operator means that all terms in the "AND group" must match some part of the searched field(s).

To search for documents that contain "OpenAPI editor" and "OAS3 Editor" use the query:

"OpenAPI editor" AND "OAS3 Editor"

OR

The OR operator divides the query into several optional terms.

To search for documents that contain "OpenAPI editor" or "OAS3 Editor" use the query:

"OpenAPI editor" OR "OAS3 Editor"

NOT

The NOT operator excludes documents that contain the term after NOT. But an "AND group" which contains only terms with the NOT operator gives an empty result set instead of a full set of indexed documents.

To search for documents that contain " OpenAPI editor " but not " OAS3 Editor" use the query:

"OpenAPI editor" AND NOT "OAS3 Editor"

&&, ||, and ! operators

&&, ||, and ! may be used instead of AND, OR, and NOT notation.

+

The "+" or required operator stipulates that the term after the "+" symbol must match the document.

To search for documents that must contain "OAS3" and may contain "OpenAPI" use the query:

+OAS3 OpenAPI

-

The "-" or prohibit operator excludes documents that match the term after the "-" symbol.

To search for documents that contain "OpenAPI editor" but not "OAS3 Editor" use the query:

"OpenAPI editor" -"OAS3 Editor"

No Operator

If no operator is used, then the search behavior is defined by the "default boolean operator".

This is set to OR by default.

That implies each term is optional by default. It may or may not be present within document, but documents with this term will receive a higher score.

To search for documents that requires " OpenAPI editor" and may contain "OAS3" use the query:

+" OpenAPI editor" "OAS3"

Overview of operators

Operator	Description	Position
AND, &&	Combines two clauses so all source files are found that match both clauses. This is a default operator of Lucene which is implicitly used if multiple clauses are just divided by space characters without explicit operator.	Between two clauses
OR, \|\|	Combines two clauses so all source files are found that match either one of them or both clauses.	Between two clauses
+	Marks the clause as "positive", i.e. all source files must match the clause. This is a default operator of Lucene which is implicitly used when clauses have no preceding operator.	Directly preceding the clause
NOT, -, !	Marks the clause as "negative", i.e. all source files must not match the clause. A query may not just consist of negative clauses.	Directly preceding the clause in case of "-" and "!", preceding the clause but divided by a space character from it in case of NOT

Escaping Special Characters

Lucene supports escaping special characters that are used in query syntax. The current list of special characters is:

+ - && || ! ( ) { } [ ] ^ " ~ * ? : \

+ and - inside single terms are automatically treated as common characters.

For other instances of these characters use the \ before each special character you'd like to escape. For example to search for (1+1):2 use the query:

\(1\+1\)\:2

If you have a word that contains any of the special characters @#$% (reserved for the search engine) then enclose the word in double-quotes to get an exact match otherwise the word will be broken up by the removal of the special characters and the hit count will be larger than expected. For example, searching for the program field A@KEYV should be changed to "A@KEYV". This statement also holds true when you want to search for a phrase that contains parenthesis, to keep the search result as accurate as possible you will need to surround the phrase in double-quotes, for example when searching for the phrase pfile(OMAPP) the result will be hits containing either the word pfile or OMAPP. When enclosing the phrase in double-quotes then the result will only be lines containing both words (in any combination or composition). If you want to make sure that the words are next to each other then you could use a proximity search such as "pfile OMAPP"~0. If you require a strict and exact match then the Xref query exact option must be enabled.

When running a full-text search with multiple words contained within double quotes and separated by spaces then the outcome may not reflect what was requested. Xref will do a search based on a breakdown of the words using the space character as the delimiter. To make sure a space is treated as an actual space you must escape the space with a backslash. For example, searching for the exact phrase "Print a line" you need to alter the text to "Print\ a\ line". If using the escape character still does not yield a satisfactory result (too many false positives) then you can also opt to use the Xref query exact match option.

NOTE. Frequently occurring special characters, for example, the double/single quotes, parenthesis, braces, and brackets are not indexed for performance reasons. Therefore it might not be possible to search for a specific pattern, for example: "the source" (i.e. including the double quotes as part of the search). Also, which special character makes it into the index is analyzer dependent, for example, the angle bracket '<' will be included by the Xref Lisp analyzer but excluded by other analyzers.

NOTE. The reserved character % is an especially tricky character and needs special handling by Xref. When this character is included in a word for example the text %eof then the text cannot be enclosed in double-quotes. There is a known issue where a % sign in the quoted text will cause the server to reject the query with a server 500 error. Also do not escape the % sign as this will cause the text to be split up into multiple words at the % sign, for example, \%bin will become the search words % and bin.

Wildcards

Lucene supports single and multiple character wildcard searches within single terms (but not within phrase queries).

To perform a single character wildcard search use the "?" symbol.

To perform a multiple character wildcard search use the "*" symbol.

The single character wildcard search looks for string that match the term with the "?" replaced by any single character. For example, to search for "text" or "test" you can use the search:

te?t

Multiple character wildcard searches look for 0 or more characters when matching strings against terms. For example, to search for test, tests or tester, you can use the search:

test*

You can use "?", "*" or both at any place of the term:

*te?t*

It searches for example "text", "test", "testcase" etc.

Wildcards do not expand over multiple words. For example, test*case; will find testcase, but test case will not be included.

Note. If you need to use a * or ? in a word or phrase as part of the actual search text then enclose the word or phrase in double-quotes.

Range Searches

Range queries allow the developer or user to match documents whose field(s) values are between the lower and upper bound specified by the range query. Range Queries can be inclusive or exclusive of the upper and lower bounds. Sorting is performed lexicographically.

full:{Boolean TO Decimal}

This will find all documents whose titles would be sorted between Boolean and Decimal, but not including Boolean and Decimal.

Inclusive range queries are denoted by square brackets. Exclusive range queries are denoted by curly brackets.

If field is not specified then a search for a specified interval is done in the default full field.

{Boolean TO Decimal}

Fuzzy Searches

Java Lucene supports fuzzy searches based on the Levenshtein Distance, or Edit Distance algorithm. To do a fuzzy search use the tilde, "~", symbol at the end of a Single word Term. For example to search for a term similar in spelling to "roam" use the fuzzy search:

roam~

This search will find terms like foam and roams. Additional (optional) parameter can specify the required similarity. The value is between 0 and 1, with a value closer to 1 only terms with a higher similarity will be matched. For example:

roam~0.8

The default that is used if the parameter is not given is 0.5.

Proximity Searches

Lucene supports finding words from a phrase that are within a specified word distance in a string. To do a proximity search use the tilde, "~", symbol at the end of the phrase. For example to search for a "OpenAPI" and "editor" within 10 words of each other in a document use the search:

"OpenAPI editor"~10

Boosting a Term

Lucene provides the relevance level of matching documents based on the terms found. To boost the relevance of a term use the caret, "^", symbol with a boost factor (a number) at the end of the term you are searching. The higher the boost factor, the more relevant the term will be.

Boosting allows you to control the relevance of a document by boosting individual terms. For example, if you are searching for

OAS editor

and you want the term "OAS" to be more relevant boost it using the ^ symbol along with the boost factor next to the term. You would type:

OAS^4 editor

This will make documents with the term OAS appear more relevant. You can also boost phrase terms and subqueries as in the example:

"OpenAPI editor"^4 "OAS3 Editor"

By default, the boost factor is 1. Although the boost factor must be positive, it may be less than 1 (e.g. 0.2).

Grouping

Parentheses can be used to group clauses to form sub queries. This can be useful if you want to control the precedence of boolean logic operators for a query or mix different boolean query styles:

+(editor OR tool) +oas3

Field Grouping

Lucene also supports using parentheses to group multiple clauses to a single field.

To search for text that contains both the word "oas3" and the phrase "openapi editor" use the query:

(+oas3 +"openapi editor")

Simple Query Examples

To find where cdxferc is used:

cdxferc

To find where cdxferc is defined:

defs:cdxferc

To find where cdxferc is referenced:

refs:cdxferc

To find C language files that have a reference to cdxferc but do not define it:

type:c +refs:cdxferc -defs:cdxferc

To find files that have a reference to cdxferc in files with the extension c:

refs:cdxferc path:c

To find files that have a reference to cdxferc in the Xref location Git_Repo_Omxfer (casing must be correct):

refs:cdxferc path:Git_Repo_Omxfer

To be more precise and get hits only from a specific path then enclose the path with double quotes, e.g. "src/mypath", otherwise dividers will be removed and you will most likely get more hits.

To search for the phrase "EXPORT short cdxferc”:

"EXPORT short cdxferc"

To find all strings that have the word cdxfer in it use the wildcard:

*cdxfer*

NOTE. If a name used in a path definition contains a space then enclose the name within double-quotes. Using path also has the restriction that you can use only a single file extension in a path statement, e.g. using path:c and path:h to restrict the query to files with the extension c or h will not work, use a language type to do this.

Xref Query Definition

As an extension to the Xref Query functionality, an Xref definition is a query template that can contain 1 or more placeholders, represented by the value '<#>', for runtime variable substitution. This construct allows a user, for example, to select a component from the TD/OMS components view and have it run against an Xref Query definition. In essence, the Xref definition describes the search query, query attributes, and which groups/locations are to be used in the query. The Xref query definition menus (Create, Update, Delete) can be accessed from the top toolbar 'Xref' menu:

Create an Xref Query Definition

In the example above a very simple query definition, with the name 'find_table_def_pfile', is created that will do a full search of the runtime provided substitute value '<#>'. This query definition can for example be used from the TD/OMS components view to locate a physical file used in an RPG PFILE keyword.

You can use a previously run query to pre-fill the dialog fields, the fields filled will match the terms and locations used in the loaded query. Using the history to load a query is very useful as you can test a definition with the Advanced Query dialog before creating it.

A query definition can contain 1 or more placeholders for variable substitution, it is up to the code logic applying the definition to replace the placeholders with actual data to be used for a search.

Update an Xref Query Definition

Select the definition to be updated from the 'Query Definition' drop-down menu, update the fields you need to change (you can also rename the query), and press finish.

Delete an Xref Query Definition

Select the definition to be deleted from the 'Query Definition' drop-down menu and press finish.

The use of Xref query definitions depends on the use of them by logic implemented in Gravity or TD/OMS. Currently, TD/OMS can utilize an Xref query definition when a component is selected in the components view. If a query definition exists then it will be available from the pop-up menu of the component. For example from the TD/OMS components view:

In the example above the component OMAPP was used as input for the query definition 'find_table_def_pfile' which in our setup yielded a single hit:

GR:Gravity/Xref/Queries/Xref Search Capabilities