Source Collectors

From Varibill Documentation
Jump to: navigation, search

Varibill is not a provisioning system but can collect usage from the various source system on which products/services were provisioned and uses that as your billing data. (Billing data / Billing records are records that relate to products/services rendered, for which a client must be billed.) This means that you will provide/configure your client's products/services as you would normally.

Varibill enables you to easily import billing data into Varibill, manually (using the Billing Data Import functionality) or automatically through the help of Source Collectors. Varibill Source Collectors allow you to integrate with your source systems through the Source Collector API to automatically collect billing data from any piece of software or hardware on which products or services are provisioned.

SC Video.png

What Are the Benefits?

  1. Routine scheduled billing data import,
  2. Reduced time and effort spent on collecting billing data manually,
  3. Reduced errors and risk of incorrect billing due to human intervention,
  4. Ability to import increased volumes of billing data,
  5. Assurance that all billing data is imported accurately into Varibill.


Existing Source Collectors

View the List Here

How Does It Work?

Your client makes use of various products/services rendered by you. Your source system then keeps track of your clients’ configuration and products/services provisioned. This information is then automatically imported into Varibill, through the help of Source Collectors, as often as you wish, so you can bill for your client’s usage.

Each Source Collector is developed, implemented and tasked to run on a schedule. (See [Source Collector Implementation] for more information)

Source Collector Topology

::Topology.jpg


Source Collector Implementation

  1. Ensure the source system has a method of communication available (API, PowerShell, SSH, Database Access, etc.).
  2. Obtain a read-only username and non-expiry password to enable it to communicate to the source system.
  3. Integrate with the Source Collector API to extract, transform and load your billing records. (Parameters explained [here]) and
  4. Configure your Source Collector on Varibill (As more fully detailed [here]).
  5. Install your Source Collector on a [Collector Server] and start importing your billing data into Varibill.

For a Source Collector to be able to "speak" to Varibill it needs the following. These fields are authenticated when posting data to your Varibill account:

  1. Point to the Source Collector API as configured on Tenant Details. We recommend that this be a configurable field when implementing your Source Collector so that this can easily be changed in future if need be.
  2. A valid Source Collector user. This user is automatically created when provisioning your Varibill account. This user appears as a normal user but has a system flag to authorise it to import source data. We recommend that the Source Collector Username and Password be configurable fields when implementing your Source Collector so that this can easily be changed in future if need be.
  3. A Source Collector registered on Varibill using the form available under Admin, Source Collectors. (See Collector Setup Attributes Explained). We recommend that the Source Collector Name and Key be configurable fields when implementing your Source Collector so that this can easily be changed in future if need be.

Download the Implementation Guidelines to assist you during your implementation process.

Please Note:

When Varibill develops Source Collectors on your behalf, we will only integrate with source systems, and versions, that is supported by the vendor of the source system.

Tip!!

When a new Source Collector is developed, and the products collected include products with a billing principle of average, discrete or maximum it is advisable that Source Collectors be developed and switched on one day exceeding your billing run, to ensure a full months’ worth of data is collected and therefore accurately billed. This also depends on your collection schedule. Ensure your time it accurately to ensure you do not miss data relating to the first day of your billing cycle. We also advise that a test import be done before the go-live date to ensure you are happy with the way the data is collected.

As a gatekeeper, we advise that the Source Collector itself send email notifications to whoever is responsible for the monitoring of this Source Collector to inform them if the Source Collector cannot communicate with Varibill for whatever reason ([Refer to Monitoring your Source Collector]). If you have elected for an on-premise Varibill environment, we recommend you refer to the Implementation Guidelines for email delivery options and recommendations.

Source Collector Server

  1. It is recommended to implement the Source Collector on a stand-alone machine (best practice), virtual or physical, that meets the minimum requirements of the selected operating system. For Microsoft, Microsoft 2012 R2 Standard, or higher, to ensure the optimal performance, availability and redundancy as required by the client.
  2. The selected operating system must support the latest JDK (Java Development Kit). It can be downloaded here.
  3. The Source Collector server should preferably have access to an SMTP server to be able to send exception emails on the Source Collector's performance.
  4. Ensure that port 80 (HTTP) and 443 (https) are open to the Varibill API (The cloud-based API is api.varibill.com)
  5. The Source Collector server must have access to the applicable source system host(s).
  6. Connectivity should be available to all source host(s)from the Source Collector machine.
  7. Ensure the source system hostname and port are accessible from the Source Collector server.


Source Collector Setup Attributes Explained

Source Name

The name of the Source Collector makes it easily recognisable, but does not influence the collection of billing data.

Status

The status of the Source Collector determines if billing data will be imported via the Source Collector or not.

Include in Validation and Billing

Select “No” to exclude any data that has been collected by this Source Collector from Billing Validation and Processing. This option does not prevent this Source Collector from collecting data.

Select “Yes” to include any data that has been collected by this Source Collector from Billing Validation and Processing.

Key

The key is the unique number that the system uses to recognise the Source Collector as a valid collector. This is used as a security measure to ensure only authorised collectors are importing billing data into your Varibill account. This key has to be provided as one of the parameters when connecting with the Source Collector API

IP Address

The IP address informs the system where the Source Collector is running from. This is used as a security measure to ensure only authorised collectors are importing billing data into your Varibill account. You can specify one / more IP addresses where the Source Collector should be able to run from. The IP address where the Source Collector is posting from will be validated by the Source Collector API.

Processing Rule

“Reject complete batch”

The system will reject the complete batch should one / more billing records fail validation.

“Reject failed records only”

Only the billing records that failed validation will be rejected and the balance of the billing data records will be imported.

Source Collector Description

The Source Collector description field can be used for internal use to identify the purpose of the collector or explain additional information pertaining to the Source Collector.

Notification Frequency

This indicates how often you would like to receive notifications on the failure-to-run-status of this Source Collector. This should be in line with the schedule of the Source Collector.

Scheduled Run Time

This indicates when the Source Collector is expected to run.

Last Run Check Time

This indicates when Varibill should determine if the Source Collector run was successful or not. If the run was unsuccessful a notification will be sent at this time to inform the notification recipients. It is recommended that this time not be the same as the Notification Interval as false notifications may be sent due to the Source Collector still being busy with its scheduled run at the Notification Interval time.

Email Recipients

Indicate the name and email address of the individuals who should receive the failure notifications for this Source Collector.


Designing the Source Collector (Source Collector API Parameters)

Source Collectors are purely a reader of the configuration and provisioned products and services of the source system. Depending on how the Source Collector is developed the Source Collector may exclude inactive products or services or may include these when collecting the data. Important! Ensure you set the Source Collector rules to only include the data you wish to bill for.

Design the Source Collector to extract configuration or usage from the source system, transform this into the format required by Varibill and load this into the Varibill database using the Varibill API.

The parameters required by the Source Collector are:

(These parameters identify the Source Collector to ensure valid posts are made to your Varibill account.)

Key

Refer to [Setup Attributes Explained] "Key". We recommend that this field be a configurable field when implementing your Source Collector so that this can easily be changed in future if need be.

Name

Refer to [Setup Attributes Explained] "Source Name". We recommend that this field be a configurable field when implementing your Source Collector so that this can easily be changed in future if need be.

(These parameters are used to post the actual billing records to Varibill. One or more records can be posted to Varibill per post.)

Record Type

(For File Upload ONLY) Each billing record must have a record type of "R". At the end of the upload file a record with record type "T" must indicate the total records ("R") expected in the file. The header for this column in the upload file is "RecordType". In the example provided, each row, except for the header row and the total row, has a record type of "R". The example file has four records, therefore the total ("T") will be "4". If an additional row were added, the value for "T" would be "5". The value for "T" must be specified in column B, in the same row as record type "T". In the example record type "T" is in row 6, therefore the value is populated in cell B6.

Client ID {ClientIdentifier}

This value will be used to link your billing record to an active contract. This value does not have to match the client's account code as it’s created in Varibill or in your accounting system. This value can be the value as extracted from the source system. You can link this value to a contract once the data has been imported using the "Client Identifiers" screen. The header for this column in the upload file is "ClientID". If a client ID does not yet exist in Varibill, and you validate your billing data (Billing Validation), Varibill will give an error for each client identifier that does not exist or has not been set up correctly. This value is never seen by your clients. The ClientID is limited to 150 characters.

Product Code {ProductCode}

This value should match the product code as it’s created in Varibill (Refer to Products). The header for this column in the upload file is "ProductCode". If the product does not yet exist in Varibill it can be crated at any time. When you validate your billing data (Billing Validation) Varibill will give an error for each product code that does not exist. The ProductCode is limited to 200 characters.

Record ID {RecordIdentifier}

The value of this field will be reflected on your client’s invoice once the client drills into the detail of a line item, i.e. view the invoice online and expand the line item. The header for this column in the upload file is "RecordID". Use a sensible description here. The Record ID is limited to 400 characters, but keep in mind that your clients would not want a paragraph per record on his/her invoice.

GUID {RecordGUID}

This value uniquely identifies the record, together with a few other fields (See Billing Record Import Status for more information). The header for this column in the upload file is "GUID". This value does not have to be a GUID / UUID [[1]], but is preferable. In the example provided a simple value such as "1" has been used. This value is never seen by your clients. The GUID is limited to 400 characters.

Last Seen Date {LastSeenDateTime}

The last seen date of a record indicates the date and (optionally) the time on which a record was seen. This date will be used when calculating pro-rated rates, where applicable.

This field must be in a valid date format and should preferably contain a timestamp (DATETIME data type e.g. 2019-01-21T08:17:36). Varibill follows the [|ISO 8601] standard which means the TZD (Time Zone Designator) must be Z or +hh:mm or -hh:mm or not provided. It is then also imperative to take the server, receiving the data, time zone into consideration. If your instance is hosted by Varibill then the timezone is GMT+02:00.

Herewith a few examples of each TZD format to follow
Example of data collected at source which has been converted into the Varibill API format using the TZD +hh:mm:</p>
  <soapenv:Body>
     <tem:records>
       <var:Record>
         <var:ClientIdentifier>7</var:ClientIdentifier>
         <var:LastSeenDateTime>2019-03-14T09:18:25+08:00</var:LastSeenDateTime>
         <var:ProductCode>43</var:ProductCode>
         <var:Quantity>188</var:Quantity>
         <var:RecordGUID>3gYSV0zLtYqiNQ8qN7PN</var:RecordGUID>
         <var:RecordIdentifier>Eisenhower</var:RecordIdentifier>
       </var:Record>
     </tem:records>
  </soapenv:Body>

Example of how the data is pulled into the Varibill database:</p>

DataExample.PNG


Example of data collected at the source which has been converted into the Varibill API format using the TZD -hh:mm:</p>
  <soapenv:Body>
     <tem:records>
       <var:Record>
         <var:ClientIdentifier>8</var:ClientIdentifier>
         <var:LastSeenDateTime>>2019-03-14T09:18:25-05:00</var:LastSeenDateTime>
         <var:ProductCode>44</var:ProductCode>
         <var:Quantity>314</var:Quantity>
         <var:RecordGUID>24ILbkJHMChvH8ofvkZE</var:RecordGUID>
         <var:RecordIdentifier>Pierce</var:RecordIdentifier>
       </var:Record>
     </tem:records>
  </soapenv:Body>

Example of how the data is pulled into the Varibill database:</p>

DataExample2.PNG

Example of data collected at the source which has been converted into the Varibill API format using the TZD "Z":</p>
 <soapenv:Body>
     <tem:records>
       <var:Record>
         <var:ClientIdentifier>8</var:ClientIdentifier>
         <var:LastSeenDateTime>>2019-03-14T09:18:25Z</var:LastSeenDateTime>
         <var:ProductCode>40</var:ProductCode>
         <var:Quantity>292</var:Quantity>
         <var:RecordGUID>ZMwfLzwQuJkUgGjxJyZ1</var:RecordGUID>
         <var:RecordIdentifier>Tyler</var:RecordIdentifier>
       </var:Record>
     </tem:records>
  </soapenv:Body>

Example of how the data is pulled into the Varibill database:</p>

DataExample3.PNG

Example of data collected at the source which has been converted into the Varibill API format without applying TZD:</p>
 <soapenv:Body>
     <tem:records>
       <var:Record>
         <var:ClientIdentifier>1</var:ClientIdentifier>
         <var:LastSeenDateTime>>2019-03-14T09:18:25</var:LastSeenDateTime>
         <var:ProductCode>6</var:ProductCode>
         <var:Quantity>453</var:Quantity>
         <var:RecordGUID>nHyNOfSxbsh5uc96qv51</var:RecordGUID>
         <var:RecordIdentifier>Van Buren</var:RecordIdentifier>
       </var:Record>
     </tem:records>
  </soapenv:Body>

Example of how the data is pulled into the Varibill database:</p>

DataExample4.PNG

If this is your first billing data import (automatically / manually) for a set of records, ensure you upload a file to indicate the date from which Varibill must start billing. As long as the client ID, product code, record ID and GUID combination remain the same for all billing data imports, for this record, Varibill will know the date from which it should start to bill.

The header for this column in your upload file is "LastSeenDate".

Quantity {Quantity}

The number of units that the client made use of.

The header for this column in the upload file is "Quantity". Be sure to import the records in the correct unit. The conversion rule selected for the Products will convert this quantity, if applicable, when calculating the line item value. This value can be supplied in number or decimal format ([precision = 18, scale = 5]). (Refer to Tax_and_Rounding for more information on how decimals are handled in Varibill).

Batch ID {batchIdentifier}

(For Source Collected Data ONLY) This field is optional and can be used to specify a unique ID per batch posted. This could be handy when a unique ID (In the form of a GUID) is available at source for a specific batch posted and you wish to tie back collected data to source batches or if you wish to group multiple posts into one batch. A batch could consist of one or multiple records. Varibill will generate a batch identifier, per post, if this value is not supplied.

Example XML Download the SoapUI project

  <soapenv:Header/>
  <soapenv:Body>
     <tem:ImportSourceDataV2>
        <tem:source>           
           <var:Key>?</var:Key>
           <var:Name>?</var:Name>
        </tem:source>
        <tem:records>
           <var:RecordV2>
              <var:ClientIdentifier>?</var:ClientIdentifier>
              <var:LastSeenDateTime>?</var:LastSeenDateTime>
              <var:ProductCode>?</var:ProductCode>
              <var:Quantity>?</var:Quantity>
              <var:RecordGUID>?</var:RecordGUID>
              <var:RecordIdentifier>?</var:RecordIdentifier>
           </var:RecordV2>
        </tem:records>
        <tem:batchIdentifier>?</tem:batchIdentifier>
     </tem:ImportSourceDataV2>
  </soapenv:Body>


Interpreting the results of your Source Collector run

<ImportSourceDataResult>
   <BatchIdentifier>E27C2D09-9B1D-48AB-93F8-1864EAE7FF3B</BatchIdentifier>
   <DuplicateRecords>0</DuplicateRecords>
   <ElapsedMilliseconds>70.7545</ElapsedMilliseconds>
   <EndDateTime>2019-01-29T07:51:28.5265303+02:00</EndDateTime>
   <ExitCode>0</ExitCode>
   <Messages/>
   <NewRecords>107</NewRecords>
   <Outcome>Successful</Outcome>
   <ProcessedRecords>107</ProcessedRecords>
   <SourceIP>::1</SourceIP>
   <SourceKey>39ECECE1-9C9D-4487-8F1F-ECEB53630E4C</SourceKey>
   <SourceName>Example Source Collector</SourceName>
   <StartDateTime>2019-01-24T11:51:20.9557758+02:00</StartDateTime>
</ImportSourceDataResult>

BatchIdentifier

This will be the Batch Identifier sent in the request. If this value was not provided this value is system generated.

StartDateTime

Start Date time of the request.

Outcome

Request outcome.

ExitCode

A negative number will indicate a failure with the error message in the <Messages> field.

Exit Code Code Description Explanation
0 Successful No further action required. The data has been imported into Varibill and is available under [Data Maintenance].
-1 UnknownError An unknown exception has occurred. Contact [Support] for assistance.
-2 InvalidSourceKeyAndIPAddress The Source Key of the Source Collector does not match the Key of the Source Collector on Varibill, and / or the IP Address from which the Source Collector is communicating has not been set as a valid IP Address for the Source Collector on Varibill, and/or the Source Name of the Source Collector does not match the name of the Source Collector as set up on Varibill. (Refer to [Source Collector Setup Attributes Explained]
-3 NoServiceAccount The Varibill account has no users flagged as a service account. Contact [Support] for assistance.
-4 MutexFail The Source Collector could not get exclusive access to process the request. Please retry later.
-5 inactiveStatus The Source Collector is set to inactive. Refer to [Source Collector Setup Attributes Explained] to change the status of the Source Collector on Varibill.
-6 billingLock The Varibill account to which the Source Collector is posting is currently busy with billing processing and cannot currently accept new data. Please retry later.

SourceKey

Source Key received.

SourceName

Source Name received.

SourceIP

IP address of the requestor.

Messages

A message from Varibill, mostly used for errors.

ProcessedRecords

Number of records processed.

NewRecords

Number of unique new records seen.

DuplicateRecords

Number of records posted that is already in Varibill.

EndDateTime

End Date Time of the request.

ElapsedMilliseconds

Time in milliseconds that Varibill took to process the batch.


Monitoring Your Source Collector

Source Collector notifications can be configured on the Source Collector setup screen on Varibill. These notifications allow you to send failure notifications to one or more individuals when Varibill has not received any data from this Source Collector at the expected interval. These notifications can be configured by updating the Notification Frequency, Notification Interval and Last run check time of each Source Collector.

In addition to this, the directory where the Source Collector was installed will contain a log file. An example XML is placed within this log file every time the Source Collector is run (successfully / unsuccessfully). This XML will provide you with some useful information for that particular run. In addition to this, the Source Collector will notify you if it runs into an exception.

When Varibill develops Source Collectors on your behalf, it is designed and developed to send email notifications from the Source Collector itself in case the Source Collector cannot communicate with your Varibill account for whatever reason (Refer to Possible failure reasons for more information). These email notifications will not reflect in your Varibill account under the Email Activity Log as they are not sent by Varibill. If you have elected for an on-premise Varibill environment, we recommend you refer to the Implementation Guidelines for email delivery options and recommendations.

Once your Source Collector successfully imported data into Varibill, this data can be viewed on the Billing Data Maintenance screen.


Source Collector Activity

A list of Source Collector activity is listed and displays amongst others the date and time processed, total records processed as well the result XML generated.

This list will only display records when the Source Collector could reach the Source Collector web service. Should an entry not be made on an expected date and time, the source server should be consulted to view the result XML and corresponding error received.

Possible Source Collector Failure Reasons

  1. The read-only user's password, reading the data from the source, has changed at the source, but not on the Source Collector.
  2. The task is not scheduled to run, or is not scheduled to run as frequently as required.
  3. The IP, from which the Source Collector is allowed to run, has been changed and hasn't been changed on Varibill.
  4. The Source Collector key has been changed on Varibill, but not on the Source Collector.
  5. The web service is not reachable from the source system.
  6. The task scheduler username and/or password has expired or changed.
  7. The directory structure relating to the Source Collector has changed.

As a first step, rerun the collector manually.


Next Steps

  • Link collected records to contracts (Client Identifiers)
  • Review collected data for correctness (Billing Data Maintenance)
  • Related

  • Frequently Asked Questions