Azure User Import

Home > Integration > Hornbill Open Integration Tools > Azure User Import

About the Hornbill Azure User Import Utility The utility provides a simple, safe and secure way to create user accounts on the Hornbill platform by synchronizing with accounts held in your Azure AD. The tool is designed to run behind your corporate firewall, connect to your Azure instance, query the required account information, transform and load into the Hornbill instance. The tool connects to the Hornbill and Azure instances in the cloud over HTTPS/SSL so as long as you have standard internet access then you should be able to use tool without the need to make any firewall configuration changes. The tool supports both the initial bulk import as well as incremental adds and updates. You can schedule the tool to run periodically to perform the import/update tasks as required. The utility employs the Azure Graph API to query the contents of Azure AD. If you would like to know more about this API and it's capabilities, please refer to the relevant Microsoft documentation: Azure Graph API Information The last utility using the Azure Graph API is version 1.4.4 (download from GitHub) As of v2.0.0 the utility uses the Microsoft Graph API instead. Please refer to Microsoft Graph API Information. Please note that you will likely need to set a different set of permissions AND generate a new ClientSecret for the changes to take effect. Prior to v2.3.0 the documentation is different and can be found here Prior to v3.0.0 the documentation is different and can be found here Open Source The Azure User Import Utility is provided open source under the Hornbill Community Licence and can be found here on GitHub

Related Articles Azure App Registration Instructions User Import TimeZone Fields Options User Import DateTime Format Options Hornbill API Keys Hornbill KeySafe Hornbill Users Hornbill Organisations

Installation Overview

Windows Installation

• Download the architecture specific latest package from GitHub
• Extract zip into a folder you would like the application to run from e.g. C:\Hornbill_Import\
• Open conf.json and add in the necessary configuration
• Open Command Line Prompt as Administrator
• Change Directory to the folder with azure_user_import.exe C:\Hornbill_Import\
• Run the command azure_user_import.exe -dryrun=true

HTTP Proxies

If you use a proxy for all of your internet traffic, the HTTP_PROXY and HTTPS_PROXY Environment variables need to be set. These environment variables hold the hostname or IP address of your proxy server. It is a standard environment variable and like any such variable, the specific steps you use to set it depends on your operating system.

For windows machines, it can be set from the command line using the following:
set HTTP_PROXY=HOST:PORT

set HTTPS_PROXY=HOST:PORT
Where "HOST" is the IP address or host name of your Proxy Server and "PORT" is the specific port number. IF you require a username and password to go through the proxy, the format for the setting is as follows:
set HTTP_PROXY=username:password@HOST:PORT

set HTTPS_PROXY=username:password@HOST:PORT

URLs to White List

Occasionally on top of setting the HTTP_PROXY variable the following URLs need to be white listed to allow access out to our network

• https://files.hornbill.com/instances/INSTANCENAME/zoneinfo - Allows access to lookup your Instance API Endpoint
• https://files.hornbill.co/instances/INSTANCENAME/zoneinfo - Backup URL for when files.hornbill.com is unavailable
• https://eurapi.hornbill.com/INSTANCENAME/xmlmc/ - This is your Instance API Endpoint, eurapi can change so you should use the endpoint defined in the previous URL
• https://api.github.com/repos/hornbill/asset-rel-import/tags - Allows the utility to self-update. Optional

Configuration Overview

Prior to configuring the .json file, it is advisable to read the following wiki page regarding Hornbill User Accounts as it will provide some context to the content on this page.

A default configuration file is provided conf.json, if a configuration file is not specified as a command line argument then conf.json must exist.

 {
   "APIKey": "", /* this is the API-key which is associated to a user in the Hornbill instance [1] */
   "InstanceId": "", /* your Hornbill instance name : not likely to change. Please note this value is case sensitive [2] */
   "AzureConf": {
       "KeySafeID": 0, /* [3] */
       "Tenant": "",
       "ClientID": "",
       "ClientSecret": "",
       "UserFilter": "startswith(displayName,'Dave')",
       "UserProperties": [ /* list all the fields that you require from Azure */
           "businessPhones", 
           "employeeId",
           "mailNickname",
           "mail",
           "givenName",
           "surname",
           "telephoneNumber",
           "department"
       ],
       "UserID": "mail",
       "Debug": false,
       "APIVersion":"v1.0", /* Azure API Version - leave as-is */
       "Search":"groups",
       "UsersByGroupID":[
           {
               "ObjectID":"Group Object ID",
               "Name":"Group Object Name"
           },
           {
               "ObjectID":"Second Group Object ID",
               "Name":"Second Group Object Name"
           }
       ]
   },
   "User": {
     "Operation":"Both", /* options : Create/Update/Both ; import actions to perform on the discovered user records */
     "UserDN": "{{.userPrincipalName}}",
     "AccountMapping":{ /* mapping of fields in the format: "Field in Hornbill": "data to insert" [4] */
         "UserID":"{{.mail}}",
         "LoginID":"{{.mail}}",
         "EmployeeID":"{{.mail}}",
         "UserType":"basic", /* (basic vs user) */
         "Name":"{{.givenName}} {{.surname}}",
         "Password":"", /* if left blank a random password will be generated [7] */
         "FirstName":"{{.givenName}}",
         "LastName":"{{.surname}}",
         "JobTitle":"",
         "Site":"1", /* if set, see also comments below on SiteLookup [8] */
         "Phone":"{{index .businessPhones 1}}",
         "Email":"{{.mail}}",
         "Mobile":"",
         "AbsenceMessage":"",
         "TimeZone":"", /* see User Import TimeZone Fields Options */
         "Language":"", /* ISO 629 in combination with ISO 3166 as per this wikipedia entry */
         "DateTimeFormat":"", /* see User Import DateTime Format Options */
         "DateFormat":"",
         "TimeFormat":"",
         "CurrencySymbol":"", /* any character */
         "CountryCode":"" /* expects ISO 3166 Alpha 2 two Character Country Code (see here) */
       },
       "Type":{
         "Action":"Create" /* options : Create/Update/Both ; setting to Basic User (as opposed to Full User) */
       },
       "Status":{
         "Action":"Both", /* options : Create/Update/Both ; on what action to change the User Account Status */
         "Value":"active" /* options : active/suspended/archived */
       },
       "Role":{
           "Action":"Both" /* options : Create/Update/Both ; on what action to change the User Role */
           , "Roles":[ /* list of Roles assigned to the users being imported */
             "Basic User Role"
           ]
       },
       "ProfileMapping":{ /* further fields [5]  */
         "MiddleName":"",
         "JobDescription":"",
         "Qualifications":"",
         "Interests":"",
         "Expertise":"",
         "Gender":"",
         "Dob":"",
         "Nationality":"",
         "Religion":"",
         "HomeTelephone":"{{.telephoneNumber}}",
         "SocialNetworkA":"",
         "SocialNetworkB":"",
         "SocialNetworkC":"",
         "SocialNetworkD":"",
         "SocialNetworkE":"",
         "SocialNetworkF":"",
         "SocialNetworkG":"",
         "SocialNetworkH":"",
         "PersonalInterests":"",
         "homeAddress":"",
         "PersonalBlog":"",
         "Attrib1":"1",
         "Attrib2":"2",
         "Attrib3":"3",
         "Attrib4":"4",
         "Attrib5":"5",
         "Attrib6":"6",
         "Attrib7":"7",
         "Attrib8":"8"
       },
       "Manager":{
           "Action": "Both" /* options : Create/Update/Both/__clear__/__clear_if_empty__ ; on what action to change the User's Manager */
           , "AzField": "userPrincipalName" /* field in Azure containing the manager's unique identifier - (default: userPrincipalName) */
           , "HornbillUniqueColumn":"h_user_id"  /* field in Hornbill to match resulting AzField to - (default:h_user_id) */
       }
     }
   }
   , "Image":{  /* Profile Image configuration section [10] */
     "Action":"Both" /* options : Create/Update/Both ; on what action to change the Image */
     , "UploadType": "AZURE" /* options : URI/URL/AZURE ; local (network) drive or HTTP(S) served image */
     , "InsecureSkipVerify": false
     , "ImageType": "jpg" /* options : jpg/png */
     , "ImageSize": "240" /* options: ""/48/64/96/120/240/360/432/504/648 see  restrictions; "" will use the default MS thumbnail size */
     , "URI": "{{.id}}"
   }
   , "Site":{
     "Action":"Both" /* options : Create/Update/Both ; on what action to change the User's Site [9] */
     , "Value": "{{.physicalDeliveryOfficeName}}"
   }
   , "Org":[ /* Organisational Units to associate the imported user with [6] */
     {
       "Action":"Both" /* options : Create/Update/Both ; on what action to add to the organisational structure */
       , "value":"{{.department}}" /* name of organisational unit */
       , "MemberOf":""
       , "Options": {
         "Type":2, /* options : 0,...,5 ; type of organisational unit, respectively: general/team/department/costcenter/division/company */
         "Membership":"member", /* options : member/teamLeader/manager */
         "TasksView":false, /* options : true/false ; If set true, then the user can view tasks assigned to this group */
         "TasksAction":false, /* options : true/false ; If set true, then the user can action tasks assigned to this group */
         "OnlyOneGroupAssignment":false
       }
     }
     , {
       "Action":"Both"
       , "value":"Great Company"
       , "MemberOf":""
       , "Options": {
         "Type":5,
         "Membership":"member",
         "TasksView":false,
         "TasksAction":false,
         "OnlyOneGroupAssignment":false /* options: true/false ; if set to true, then a user can only be associated to a single group at any one time */
       }
     }
   ]
 }
 }

1. A valid API key needs to be created against a Hornbill user account with enough rights to create and update user accounts. Details on how to create an API key can be found here.
2. The Instance ID (also referred to as the instance name) can be found in the URL used by your organisation to access your Hornbill instance i.e. https://live.hornbill.com/instanceid/ (case sensitive).
3. You have two options here:
   1. create a KeySafe record of type "oAuth 2.0", set Client Id, Client Secret as expected, the **Tenant ID** should go into the field labeled API Endpoint (final one listed, as of this writing). The KeySafeID can be obtained from looking at the URL of the KeySafe record https://live.hornbill.com/instanceid/security/keysafe/#KeySafeID/. Using a KeySafeID other than 0 will override the following three fields.
   2. Tenant, ClientID and ClientSecret can be set in the configuration file, however using KeySafeID is considered more secure as it can be matched to specific API Keys
   3. There are relevant Microsoft instructions on-line on how to obtain the ClientID and ClientSecret from within Azure - our quickstart version. We have found that the following permissions need to be granted within Azure, though these could differ for yourselves, so please rely on your own expertise. Application permissions on: Group.Read.All, GroupMember.Read.All, Team.ReadBasic.All, TeamMember.Read.All, User.Read.All. Delegated permission on User.Read. The permission settings need confirming.
4. The fields are quite self-explanatory and part of the "Details"-section (as opposed to the "About"-section [5]), most can be left as-is. The mapping is done with some templating of the format {{fieldname}}. One can be a little adventurous, for instance "{{.mgrfirstname}} - {{.mgrlastname}}" puts a space, hyphen and space between the person's manager's first and last name. One can use this to prefix or suffix the values coming from the database or indeed to set a static value (as done for example with userType and Attribute under OrgLookup for the company (Type:5)).
5. The same holds here as for [4], this is regarding the "About"-section of the user's details.
6. this is a non-ordered list of the organisation structure - it allows one to associate the imported user to one or more levels within the organisation. The delivered configuration file will associate each imported user with EACH of the three discernable levels (company, department and division) - depending on your requirements regarding the availability of services and such, you will likely want to manipulate this section and perhaps only leave behind the "department" level (Type: 2) with the TasksView and TasksAction set accordingly. Please note that this import only adds and not REMOVES any association.
7. The password field should be left empty as the utility generates a secure password that adheres to the User Password Policy as specified on your Hornbill instance. This password will only be temporary as the user should use the "Forgot Password" link available on the Hornbill Login Screen to reset their password the first time they navigate to your Hornbill instance.
8. "Site" - Recognises a corresponding Hornbill site ID. E.g. "Site":"1" - The value of Site should be numeric. As an alternative, the import configuration provides a "Site Lookup" section (outlined in a later section) which can make a site association based on the contents of a directory attribute.
9. If a lookup action is not needed, remove the action type so only empty quotes are left i.e. "Action":""
10. "Image" - by default this will take the thumbnail image. IF .id does NOT work for the "URI", please try .userPrincipalName instead.

Filtering

There are two methods of filtering users that you can configure with this tool. They are both defined in the AzureConf section of the configuration.

To import all direct User objects within one or more Azure Groups:

• AzureConf > Search : set the value of this parameter to groups
• AzureConf > UsersByGroupID : this is a JSON array of Groups to return Users from:
  • ObjectID : the Object ID of the Group you want to return direct member Users from;
  • Name : The Name of the Group

Using an Azure AD filter to find Users to import:

• AzureConf > Search : set the value of this parameter to users
• AzureConf > UserFilter : define an Azure filter to search for User objects. If not defined, then all User objects from your Azure AD will be returned.

Fields

These fields are those which Azure AD recognise as part of an account (eg givenName) they match LDAP variables quite nicely. However, please keep in mind that although for instance multiple email addresses can be set in Azure, only the main one in mail can be used (unless one makes amendments to the script)

Associating a Site to Hornbill User Accounts

The DB Import utility has the ability to associate a Hornbill Site record to a user account based on the contents of a field. This is achieved through a "Look-up". The Look up mechanism is quite simple and works in the following manner.

1. The import reads the fields (template rules work) that is specified in the "value" field. In the example shown, the site field is used.
2. It takes the content and tries to identify if there is an existing site record in Hornbill with a name that matches the value of the site. e.g. if the site field contained "Brussels", the import would look for a Hornbill Site record with the name "Brussels".
3. If a match is found, the import will associate the user to the site.
4. If no site record is found, the import will move onto the next user.

i.e. The name of the Site record in Hornbill must match the value of the directory attribute specified. More on Hornbill Sites can be found here: Sites

Associating a Group to Hornbill User Accounts

The DB Import has the ability to associate a Hornbill Group to a user account based on the contents of a fieldname. This is achieved through a "Look-up". The Look up mechanism is quite simple and works in the following manner.

1. The utility reads the attribute that is specified in the orgLookup section. In the example shown, the department field is used.
2. It takes the content and tries to identify if there is a Hornbill Group that exists with a name that matches the value of the field name. e.g. if the department field contained "Accounting", the utility would look for a Hornbill Group called "Accounting".
3. If a match is found, the import will associate the user to the group.
4. If no Hornbill organisation is found, the import will move onto the next user.

i.e. The name of the Organization(Group) in Hornbill must match the value of the database field. More on Hornbill Organisational Groups can be found here: Organisation Structure

User Manager Mapping in Hornbill

Both Hornbill and Azure can store a manager relationship between two users.

The utility will determine the manager for a user as it is stored in Azure and use the field given in AzField to match this against a user within Hornbill using the field defined in the HornbillUniqueColumn.

In instances whereby the userPrincipalName is used as the Hornbill User ID, the defaults should suffice.

IF, for instace, Azure's employeeId is used as User ID in Hornbill, then AzField should be set to employeeId.

The manager link is handled by the ProfileMapping part in the Hornbill API - so the link will be set on: Both, Create or Update based on what the regular Account is doing.

IF you do NOT want the Azure Import to set a manager, just do not use any value in the Action parameter. If you want to activate the manager mapping, then any of the three Both/Create/Update values will suffice.

You have two additional Actions:

• __clear__ - this will remove the managers from each individual found in the Azure search
• __clear_if_empty__ - will remove the managers from each individual found in the Azure search who do NOT have a manager

Preparing to Run the Import

Ultimately, the executable will be scheduled in Windows task scheduler (see later) but to test, gain confidence, and perform the initial upload of users the utility can be executed from a command prompt window on an ad-hoc basis. The command used to execute the import can contain a number of command line parameters.

• dryrun - Defaults to false - Set to True and the XMLMC for Create and Update users will not be called and instead the XML will be dumped to the log file, this is to aid in debugging the initial connection information.
• file - Defaults to conf.json - Name of the Configuration file to load
• zone - Defaults to eur - Allows you to change the ZONE used for creating the XMLMC EndPoint URL: https://{ZONE}api.hornbill.com/{INSTANCE}/
• workers - Defaults to `3` - Allows you to change the number of worker threads used to process the import, this can improve performance on slow import but using too many workers have a detriment to performance of your Hornbill instance.
• debug - Defaults to false - outputs extra information to the log to help with debugging issues.

Testing Overview

There is no substitute for hands-on experience when becoming familiar with the Hornbill import utilities.
The Azure User import accepts and understands a number of "Command Line Parameters" that can be used when running the utility from the command line. The most important one for testing is the -dryrun=true command. When this is specified, no information will be written to Hornbill and it allows you to confirm that the configuration is correct and that a connection to your directory server can be established. A dryrun still outputs a log file which provides you with an opportunity to review and understand any error messages that may occur.
Below are some high level steps to help you build confidence in your configuration:

1. In the configuration, specify an "UserFilter" to target a single user object. (Its good practice to initially test on a single, or small set of, user objects as this allows the dryruns to complete quicker and there is less log content to sift through).
2. Perform a dryrun (by executing the utility along with the -dryrun=true command line parameter).
3. Review cmd output and log file for errors
4. Check against "Common Error Messages" listed on the wiki and take action to rectify where necessary.
5. Continue with dryrun tests until you are happy that all the errors are accounted for.
6. Perform a live import with this single user object still specified i.e. set -dryrun=false
7. Review user account in Hornbill and check all user properties are as expected i.e. email contains an email address etc.
8. Adjust conf file user property mappings as necessary
9. Loop through steps 6 - 8 as many times as is necessary until you are happy with the information being transported into the Hornbill user account properties.
10. Amend the "UserFilter" and/or the scope of the "Search" variable to target the user objects required for a full import.
11. Perform a dryrun
12. Review cmd output and log file for errors
13. Check against "Common Error Messages" listed on the wiki and take action to rectify where necessary.
14. Continue with dryrun tests until you are happy that all the errors are accounted for.

Example command line: azure_user_import.exe -dryrun=true

What Hornbill Roles are needed for the Import to Complete Successfully?

A default role is delivered with Hornbill that is designed to be used in conjunction with our range of user import utilities. The security role is called User Import and has all the necessary rights to import / update user properties.

As you may now be aware, every action within Hornbill must be performed in the context of a user account. As well as the chosen user account possessing the "user Import" role which facilitates the importing of the user accounts and updating of the user properties, this user account must posses the roles that you are associating to imported user accounts via the import utility. The above comment about roles is referring to Hornbill's security model when it comes to associating roles to user accounts, which is: Hornbill is designed to only allow the association of roles if the User who is performing the assignment of a particular role already possess the same system/application rights among the roles that they themselves possess. This security measure prevents you inflating your own rights or giving a user more rights than you have yourself.

i.e. in addition to the "User Import" role, any roles you try and assign to the user accounts being imported must be assigned to the user account logging in and running the import.

API Key Rules

This utility uses (API keys):

• activity:profileImageSet
• admin:keysafeGetKey
• admin:sysOptionGet
• admin:userAddGroup
• admin:userAddRole
• admin:userCreate
• admin:userDeleteGroup
• admin:userProfileSet
• admin:userSetAccountStatus
• admin:userUpdate
• data:entityAddRecord
• data:entityUpdateRecord
• data:queryExec
• session:getSystemLicenseInfo

Troubleshooting

Logging Overview

All Logging output is saved in the "log" directory which can be found in the same location as the executable. The file name contains the date and time the import was run Azure_User_Import_2015-11-06T14-26-13Z.log

Common Error Messages

Below are some common errors that you may encounter in the log file and what they mean:

• [ERROR] Error Decoding Configuration File:..... - this will be typically due to a missing quote (") or comma (,) somewhere in the configuration file. This is where an online JSON viewer/validator can come in handy rather than trawling the conf file looking for that proverbial needle in a haystack.
• [ERROR] Get https://api.github.com/repos/hornbill/goAzure2HUserImport/tags: dial tcp xx.xx.xx.xx:xxx: ........ - this most likely indicates that you have a HTTP proxy server on your network between the host running the executable and your Hornbill API endpoint. Ensure the http_proxy environment variable is set (See the section on "HTTP Proxies" for more information) and that the proxy is configured to allow this communication.
• panic: runtime error: invalid memory address or nil pointer deference [recovered]... - this error is suggesting an incorrectly specified attribute in the conf file. Where information is being obtained from a directory attribute, the attribute must be in the following format: {{.directoryAttributeName}}
• [ERROR] Unable to Create User: Invalid value for parameter '[parameter name]': The text size provided (31 characters) is greater than the maximum allowable size of 20 characters for column [column name] - the contents of your directory attribute exceed the maximum number of characters that can be placed in the Hornbill database column.
• [ERROR] Unable to Create User: The value in element <userId> did not meet the required input pattern constraints. at location '/methodCall/params/userId' - the user id contains characters that are not allowed. The User Id should be made up of alphanumeric characters. Full stops (.) and underscores (_) are also supported.
• [ERROR] Unable to Create User: [usedID] Error: The specified handle [Display Name] is already in use - By default, the "Handle" (Hornbill Display Name) must be unique. This error suggests a user account already exists in Hornbill which is using this handle. The duplicate-handle validation can be disabled via a setting found in Hornbill Adminsitration under "Home > System > Advanced Settings" and filtering for "api.xmlmc.uniqueUserHandle.enable"
• [ERROR] Unable to Update User: Invalid value for parameter '[parameter name]': Error setting value for column '[column name]'. bad lexical cast: source type value could not be interpreted as target - this error is indicating that the contents of your directory attribute are in a format that is not compatible with the type of the Hornbill database column. For example, you will get this when trying to place text into a database field that is of type "INT" (accepts integer values only).
• [ERROR] Unable to Load LDAP Attribute: '[LDAP attribute name]' For Input Param: '[Hornbill Parameter name]' - When the import utility is unable to load a particular LDAP attribute, this means that the attribute field in your directory does not contain a value. This error will not prevent the user account being created or updated in Hornbill and can be considered more as a warning rather than an outright failure or problem.
• [ERROR] Unable to Set User Status [status name]: You have reached your user subscription limit of [xx], you will need to expand your subscription level if you wish to add more users - The utility is trying to update the user status of an existing user account from an inactive status (i.e. "archived" or "suspended") to "active" however in order for this to be successful you must have some subscriptions available.
• [ERROR] Unable to run import, a previous import is still running - this can occur if the previous import failed to complete. Perform a manual (non-scheduled) run of the import from the command line including the argument "forcerun=true". Future imports should now run without issue.

Error Codes

• 100 - Unable to create log File
• 101 - Unable to create log folder
• 102 - Unable to Load Configuration File

Scheduling Overview

Windows

You can schedule .exe to run with any optional command line argument from Windows Task Scheduler.

• Ensure the user account running the task has rights to Azure2UserImport.exe and the containing folder.
• Make sure the Start In parameter contains the folder where the executable resides in, otherwise it will not be able to pick up the correct path. eg: