GR:Gravity/Views/ConfigurationAdmin/MailHandler/Configure
Introduction
The Gravity mail-handler is a Gravity extension that allows users to directly respond to a Gravity sent mail by responding with a comment or by creating a new item. The option to respond to a Gravity email is available to any licensed Gravity user. When the mail-handler extension is active it will regularly check for emails arriving in the configured mail box. An email will be processed if the received email is from a known Gravity user at which point either a new item will be created or a comment will be added to an existing item. The last will occur if the email subject contains an identifying item tag. The tag also known as the Gravity subject descriptor is a short id describing an existing Gravity item, the short id is enclosed in double less-than (<<), greater-than signs (>>), for example <<37,id=2610deaf#5757500>> or <<8f0ba2f2#7457184>>). If the reply email contains attachments then these attachments will also be included in the process and be added to the newly created or existing item.
Licensing
The mail-handler is a licensed module. The Gravity base license will contain information pertaining to the availability of the mail-handler module, i.e. how may users can send email to Gravity that will be processed by the mail-handler. The mail-handler license is the number of Gravity enabled mail-handler users allowed to use the mail-handler functionality. The number should be at least equal to the number of enabled Gravity mail-handler users. A mail-handler user does not need to be registered to a license the user only needs to be enabled (note that a user that does not have a license cannot login to Gravity). if the number of enabled users exceeds the mail-handler license count then an email message will be sent indicating that the base license does not suffice with regards to the mail-handler license setting.
Users bound to a named or floatig license will always be allowed to use the mail-handler functionality as long as the mail-handler has been correctly configured and has not been disabled through the Gravity base license.
How emails are processed
Emails are read from the configured mail box at a 5 minute interval. The mail-handler will use the email subject as a comment or item subject (the last if this concerns a new item), all plain text information in the email body will be extracted and added as the item comment or the item description (the last if this concerns a new item). To avoid potential cross site scripting (XSS) issues any hyperlink will be transformed to a text only format where the hyper-link will be visible between parenthesis. Also any email attachments will be added as item attachments.
If an email is successfully processed then the email will be deleted from the mail box. If processing of an email failed then the sender will receive an email message with a reason why processing failed. The same email message will also be sent to the configured ‘Inform user’. An email that was not successfully processed will also be deleted from the mail box but a copy of the email will be saved in the server’s directory .\work\mail\handler, the copy will be stored as an eml file (viewable by Outlook Express as well as some other client mail programs).
For an email that, for whatever reason, could not be successfully processed by Gravity a reply email will be sent to the user and Gravity admin detailing the reason why Gravity did not process the email.
Setting up the mail-handler configuration
The mail-handler works from a mailbox account specifically made available for the purpose of automatically handling Gravity email replies. The credentials for the mailbox account are to be set in the IMAP/POP3 security section of the form.
Add Subject Descriptor
For the mail-handler to process emails as comments to existing items then this flag must be enabled. This option directs the mail event handler (responsible for sending Gravity emails) to inject a Gravity subject descriptor (<<...>>) into a mail’s subject. If the subject descriptor is not set, and the mailbox server has been configured, then the mail-handler will treat all in-coming mail as new items to be added to an application (for the configured default application).
Mail server settings
Mailbox Server
The host name or IP address of the mail server. If left empty then no attempt will be made to process emails in the user’s mailbox. Note that if the Gravity client or server is configured with the java property -Djava.net.preferIPv4Stack=true so that only a DNS name or an IPv4 type address can be used to configure the mail server. If there is a company proxy involved and there is no SMTP relay server available, then Gravity needs to be configured using proxy properties. The following java properties can be set to support a mail proxy configuration:
-Dmail.client.proxy.protocol=[http | https] -Dmail.client.proxy.host=[ip4-address | DNS-name] -Dmail.client.proxy.port=[0-65535]
Mailbox Server Port
Optionally set an overriding mail server port else leave empty for the mail system to decide. Well known ports are 143 (IMAP plain), 993 (IMAP SSL/TLS, 110 (POP3 plain) and 995 (POP3 SSL/TLS).
Mailbox Folder
If 'INBOX' is not the user’s default mailbox folder (from which email messages are to be read) then provide the name of the folder to be used.
Mailbox Protocol
Set the protocol to be used when reading mail from the configured user’s mailbox. If you do not know which protocol to use then you should consult your mail administrator to find out how a user’s mailbox can be access by an email client. The default protocol is IMAP.
IMAP/POP3 Security
Use TLS
Enable this option if the TLS (Transport Layer Security) protocol is to be used for a secure connection with the Mail server. This is the preferred way to setup a secure connection with the mail server. If possible use this option instead of 'Use SSL'.
Use SSL
Enable this option if the SSL protocol is to be used for a secure connection with the Mail server.
User Name
This is the account name that will be used to retrieve mail messages (i.e. from this user's mail box). To make sure a reply to a Gravity email goes to the configured user’s inbox, this name must match the Email Address Prefix of the Mail Client configuration as the .
Using a shared mailbox
If you are using an Exchange or Outlook 365 shared mailbox as the inbox for Gravity mail then you will have to setup the mail user differently. For a shared mailbox you have to provide the unique combination of a fully qualified mail user combined with the shared mailbox alias. The correct syntax is '?user?@?domain?\?shared-mailbox-alias?, for example 'user@example.com\info'.
Password
The password belonging to the above provided account name.
Advanced TLS configuration options
We provide a number of IMAP/POP3 security override options that allows you to fix possible server connection errors when dealing with a mail server that does implement that latest security standards or surpasses the standards used by Gravity. All the properties described here need to be set in the client’s or server’s configuration file.
mail.tls.start.required=[true | false ]
Is a configuration property that can be used to override the starttls requirement when ‘use TLS’ has been selected. Starttls is a security option that ensures that session encryption starts before any login data is exchanged. If a mail server cannot handle the STARTTLS command then by default the connection with the mail server will fail. To support a mail server that can handle a TLS connection but not the STARTTLS command, then set the above configuration property to false. By default this option is true.
mail.tls.check.server.identity=[true | false ]
Is a configuration property that can be used to override the ssl.checkserveridentity property of the JavaMail API when TLS is enabled. You may need to set this setting to true when the certificate returned by the mail server does not contain the same host name as used in the configured 'Mailbox Server:'. For example when accessing a mail server using its ip-address instead of the mail server's DNS name (as set in the certificate) will most likely result in a host validation exception. Setting the above configuration to false will force the acceptance of the host certificate even though the host names do not match. By default this option is true (host names will be checked).
mail.ssl.session.protocol=[ TLSv1.1 | TLSv1.2 | TLSv1.3 ]
Is a configuration property that can be used to override the session protocol used for the SSL/TLS socket connection with the mail server. Here you have the option to downgrade (SSLV3) or upgrade (TLSv1.2) the default used session protocol (default is to negotiate for a TLS protocol). The selectable property values are the protocols considered (reasonably) safe as SSLv3 being the weakest and TLSV1.2 the strongest. Setting SSLv3 might be required when communicating with an outdated mail server implementation. By today's standards TLSv1.2 (or above) is the most secure and preferred setting, one that nowadays is the minimum Microsoft Office 365 will accept.
OAuth2 Authentication
As of version 8.0.2 Gravity supports the use of OAuth2 for Outlook (Microsoft Office 365) and Gmail. For OAuth2 to successfully be used with these mail servers it is necessary to provide an access token when making a connection to one of these mail servers. As the access token is very short lived (expires after 1 hour) it is not very useful to set the access token as part of the mail handler configuration. Instead the user must provide basic client information so that Gravity can get an access token from the OAuth2 provider (Outlook or Gmail) and pass this on to the mail server. In this setup the user will only have to provide client information once or when the client information is revoked. Depending on the OAuth2 provider, different client information must be provided and set in the appropriate mail handler fields.
Outlook:
client_id -> User Name, client_secret -> Password, tenant_id -> Oauth2 Data
Gmail:
client_id -> User Name, client_secret -> Password, refresh_token -> Oauth2 Data
For example:
Each provider has its own way on how to setup a mail client for automated mail reading of a mail box. Confer with your mail administrator on setting up and acquiring the necessary OAuth2 client information.
Note, the Oauth2 provider for Microsoft uses the host login.microsoftonline.com as the login server from which to get an access token. The certificate of this host is problematic for the Java runtime as it causes an SSL peer verification exception, i.e. the host name login.microsoftonline.com does not match the certificate subject provided by the peer which is stamp2.login.microsoftonline.com. The SSL exception fails the call to the login server and no access token is received. To get around this problem you need either to add the microsoft server's certificate to Gravity's certificate directory (.data/work/certificates) or add ssl.trusted.hosts=login.microsoftonline.com to the Gravity configuration file. Both the Gravity client and server need to be updated. For Gravity servers (as of 8.0.2), they now include the certificate for the Microsoft online login site. This certificate is automatically pulled in from a separate certificates directory in the installation directory of the server. The certificate in this directory will not override a user installed certificate (in .data/work/certificates) of the same name. Note that the Microsoft certificate has a short lifespan of 1 year, you still might need to manually update the certificate if you do not upgrade Gravity before the certificate expires. You can view (expiration date) and add a certificate through the Remain certificates preference page. For the Gravity Admin client, the Microsoft certificate is available from the certificates directory in the client's installation directory.
Note, when renewing the OAuth2 information after a secret has expired, for Outlook it is necessary to create a new secret key from the Azure Portal -> Manage -> Secrets & Certificates. Select the appropriate app and from Client Secrets, create a new Client Secret. From the created values you need to copy the secret value and paste this into the password field of the OAuth2 configuration.
Accepted Mail Domains
Accepted mail domains denotes the network domains from which emails can be received and of which the emails are to be processed by the mail-handler. Essentially this is a filter to include and accept emails originating from a configured domain or sender. Optionally provide a comma separated list of mail domains or email senders, for example '@remainsoftware.com to accept all emails coming from the domain remainsoftware.com or to accept a single user, set for example: user.name@gravity.zone. For those emails that have been rejected, because of a domain or sender mismatch, an email will be sent to the sender and configured admin explaining why the email was not processed by Gravity.
If this option is set with the value '*' then then all messages will be processed regardless their origin. If this option is left empty then only the mails of the company domain (as set in E-mail Suffix) will be accepted for processing.
Note: To exclude mail from the company domain will require an entry in the 'Excluded Mail Domains'.
Excluded Mail Domains
Excluded mail domains denotes the network domains from which emails received will NOT be processed by the mail-handler. Essentially this is a spam filter to exclude emails of which the sender matches a configured domain or sender. Optionally provide a comma separated list of mail domains or email users, for example '@remainsoftware.com to exclude all email coming from the domain remainsoftware.com or to exclude a single user, set for example: user.name@gravity.zone.
Note that this option takes precedence over the option 'Accepted Mail Domains'. In this setup setting the value '*' to exclude all domains will effectively block all incoming mail from being processed by the Gravity mail handler (i.e. emails will be disregared and deleted).
In contrast to 'Accepted Mail Domains', if email processing fails because the sender was matched to an excluded domain or sender then only a warning message will be written in the server log, no reply mail will be sent to the sender or the Gravity admin.
Filter Auto Reply
Provide a regex that will be used to exclude mail messages from being processed based on what is contained in the mail subject. This might be necessary to recognize for example an auto-reply or out-of-office message not recognized by the internal filters used by Gravity. If an auto reply message is not caught by Gravity then this can cause a rebound effect between Gravity and a Gravity mail recipient.
As a first line of defense Gravity will filter auto reply (or out of office) messages by looking at the mail headers and excluding mails based on what's in the headers. If a mail client does not set a mail header to indicate an auto reply as part of an auto reply message, then it is custom to set a specific message in the mail subject to indicate that the reply was automatically generated. By default Gravity does some filtering based on subject content but this will not cover all auto reply variants. Currently Gravity uses the following regex strings to filter on subject:
'^Out of Office', '^Auto:', '^Automatic reply', '^Automatisch antwoord', '^Abwesenheitsnotiz', '^Risposta Non al computer', '^Auto Response', '^Fuori Sede', '^Respuesta autom\\u00e1tica' and '^R\\u00e9ponse automatique'.
Note. When using a special language related character in the regex, use the known Unicode escape sequence for that character, for example the character à is to be encoded as \u00e0. Also make sure the regex is compatible with the standard Java regex implementation.
Strip Quote Reply
Provide an additional custom quote reply regex that will be used to recognize the starting line of text to be removed from the mail content. The custom regex can be used to not only strip old replies but any other part of an email that is not to be included in the final result of a Gravity comment or item description.
Note that the internal mail-handler reply stripping is only applied to an email that is a comment, it will not be applied to a new email (i.e. one that will result in a new Gravity item being created). If you also need to strip new emails then you can use the custom quote reply regex which allows content stripping to be applied to new emails.
Essentially the strip quote reply mechanism will try to find a line that matches a configured regex (internal and custom). If a matching line is found then content will be removed from the matching line to the bottom of the mail. The strip quote reply mechanism will be applied to comments and new items being processed by the mail-handler.
Supplying a custom quote reply regex might be necessary if the default Gravity reply strip mechanism does not deliver a satisfactory result. The custom quote reply pattern gets the first shot at stripping if the custom pattern does not yield a match then the internal reply patterns are applied (on comments only).
The value for the custom quote reply must be a valid java regex for example:
\s+\|
If any of the following characters are used in the regex:
\.[]{}()<>*+-=?^$|
then these regex meta characters must be escaped with a single back slash '\'.
For example in the text below we want to strip the signature in a message, in this case everything below line 'Carol Apple | Team Lead Application Development' needs to be rmoved. As the line indicator we use the pipe character, i.e. '|'. Because the pipe character is a regex meta character we need to escape that character in the regex as '\|'. The regex then translates into 'find the first line with a pipe character'. To make the match more precise we could also use \s+\|, which translates into 'find the first line with a pipe character prefixed with 1 or more white space characters'.
Hi, Please work on this as soon as possible.
Carol Apple | Team Lead Application Development Lofty Software Painters Ave, 1234PC Nieuwegein, Netherlands +31306115111
The result of applying the strip quote reply '\s+\|' to the above text will be:
Hi, Please work on this as soon as possible.
Thus dropping the senders contact information.
Note. When using a special language related character in the regex, use the known Unicode escape sequence for that character, for example the character à is to be encoded as \u00e0. Also make sure the regex is compatible with the standard Java regex implementation.
By default the Gravity mailing system will inject a (hidden) line indicator as part of a comment mail (when sending a comment). This hidden line indicator can be used to strip old replies out of a comment if any previous quote reply (internal or custom) mechanism does not yield a match.
Strip Quote Reply
Provide an additional custom quote reply regex that will be used to recognize the starting line of text to be removed from the mail content. The custom regex can be used to not only strip old replies but any other part of an email that is not to be included in the final result of a Gravity comment or item description.
Exclude Inline Images
With this option enabled, images in the mail body will not be extracted and added as attachments to the item. This option is usefull if you experience receiving too many nonsensical image attachments (logos, signatures etc.) in your items. A side effect of enabling this option is that the mail-handler cannot distinguish between sensical and nonsensical images all images in the mail body will be excluded even those the user deems important.
Default Application
Select the application that will be the target for the creation of new items. This will apply to emails sent by a Gravity user but of which the email subject does not contain the Gravity subject descriptor (<<…>>).
Default Item Type
In addition to an application it is also necessary to choose a workflow Item type. The item type determines the kind of item to be created within the application.
Example mail-handler configuration
Note that the Email Address Prefix and the mailbox user name are the same. Also that the E-mail suffix ‘@remainsoftware’ will automatically be added to the list of accepted mail domains.
Troubleshooting
General
If the server response is something like 'No login methods supported' then the most likely reason is that the mail server does not accept clear connections, try a configuration with the 'Use TLS' or 'Use SSL' is set.
If an error occurs during the testing of the Mail-handler then it might be helpful to bring the Gravity console log to the foreground. Errors logged will be more detailed than the message displayed in the Mail configuration view, see Gravity Log viewer
Microsoft Exchange & Outlook 365
If you are having issues with an IMAP/TLS setup of Exchange or Outlook 365 then try switching to POP3. Select the POP3 protocol and select 'Use SSL', this combination may work for your particular Exchange or Outlook server implementation. Do not select 'Use SSL' if you know your mail server does not support SSL certificates.
If you are using the IMAP protocol against an Exchange 2010 server then there is an issue where the mail body is not being returned by the mail server. This results in empty comments or descriptions in Gravity. For more information follow this link Message has empty body
Error Messages
'* BYE JavaMail Exception: java.net.SocketException: Connection closed by remote host'. This message can be returned if the password set in the configuration no longer matches the user's outlook password.
Mail server SSL certificates
If the mail server certificate is being rejected then take a look at solutions available in Gravity to possibly solve the problem, see Gravity and SSL certificates