Create an Automatically Updating Cumulative History Dataset

It is easy in LityxIQ to create a dataset that contains an ongoing cumulative history of records, and optionally automatically refreshes itself.  For example, you may be required to maintain a history of all processed records, such as an ongoing campaign contact history.  You want that history to automatically update (a cumulative update) when a new campaign file is processed.

The three steps are simple and are described below.


Step 1: Create View that will maintain the Cumulative History and Populate with Initial Data

Follow typical steps for creating a view as found here:

In the example below, the dataset holding the cumulative history is "03b Cumulative Processing Metadata".  This is the new view we created.

In the Incoming Data section for this View, select the other (or potentially multiple other) datasets from where new records to add the cumulative history will be found.  In this example, that second or "other" dataset is "03a Today's Processing Metadata".

You may also want to use a filter on this Incoming Dataset if you only want to pre-populate the cumulative history with certain records.  Then save and execute the view.  When finished, this "03b" dataset will have an initial set of records that will start off the cumulative history tracking.


Step 2: Modify the View to Include the Dataset Itself as another Incoming Source

Now go back to edit this view and add a second incoming dataset, namely, this "03b" dataset itself to the list of Incoming Datasets.  The order of the datasets in the list will not matter, so you can put either one first.

For the second dataset from where we are receiving new records ("03a" in this example), ensure that you have checked the Required for Data Refresh checkbox on the Dataset and Variables tab.  This should be checked by default, but it is worth being sure.  As noted above, you may often require a filter to be set so that only the appropriate subset of records are added to the cumulative history.

There is no reason that you cannot also perform other processing steps in this view.  For example, it would be common to create some new fields to reflect other information you want to save as part of the cumulative history.  When ready, save this View.  Notice that the effect here is that when this view runs, it will take its own records at that time, and add to those records anything from the other dataset from which fresh records are being pulled.


Step 3: To Automate the Refresh Process, Put it on an Upon Data Refresh Schedule

This step is only required if you want this cumulative history to always automatically update itself when new records are ready to go.  If you don't do this, you will have to execute this dataset manually whenever you want to refresh the history.  To setup automation, just follow techniques for putting the execution on the Upon Data Refresh schedule, found here:



Now, the next time the other dataset ("03a") has been updated, this cumulative history ("03b") will automatically run.  This process will continue automatically until you remove the Upon Data Refresh schedule.