It is easy in LityxIQ to create a dataset that contains an ongoing cumulative history of records, and optionally automatically refreshes itself. For example, you may be required to maintain a history of all processed records, such as an ongoing campaign contact history. You want that history to automatically update (a cumulative update) when a new campaign file is processed.
The three steps are simple and are described below. In this example, we will have a dataset named "Daily Data Feed" that will represent fresh data each day. Our cumulative history dataset will be named "Cumulative Data History".
Step 1: Create a Derived Dataset that will maintain the Cumulative History, and Populate it with initial data
Follow typical steps for creating a derived dataset as found here: https://support.lityxiq.com/983183-Creating-a-Dataset-View.
In the Incoming Data section for this dataset, select the dataset with the daily data feed, assuming it has already been created and populated with data.
You may also want to use a filter on this Incoming Dataset if you only want to pre-populate the cumulative history with certain records. When ready, save and execute the dataset. When finished, this cumulative history dataset will have an initial set of records that will start off the cumulative history tracking.
Step 2: Modify the Cumulative Dataset to Include Itself as another Incoming Data Source
Now go back to edit this dataset's settings and add a second incoming dataset, namely itself to the list of Incoming Datasets. The order of the datasets in the list will not matter, so you can put either one first.
For the Daily Feed dataset, ensure that you have checked the Required for Data Refresh checkbox on the Dataset and Variables tab. This should be checked by default, but it is worth being sure. As noted above, you may often require a filter to be set so that only the appropriate subset of records are added to the cumulative history.
There is no reason that you cannot also perform other processing steps in the settings for the Cumulative History dataset. For example, it would be common to create some new fields to reflect other information you want to save as part of the cumulative history. Notice that the effect here is that when this derived dataset executes, it will take its own records at that time, and add to those records that are in the fresh daily feed dataset.
Step 3: To Automate the Refresh Process, Put it on an Upon Data Refresh Schedule
This step is only required if you want this cumulative history to automatically update itself anytime the daily feed has been refreshed. If you don't do this, you will have to execute this dataset manually whenever you want to refresh the history. To setup automation, just follow techniques for putting the execution of the cumulative history dataset on the Upon Data Refresh schedule, found here: https://support.lityxiq.com/882325-Using-the-Scheduling-Dialog.
Finished
Now, the next time the daily feed has been updated, the cumulative history will automatically refresh as well. This process will continue automatically until you remove the Upon Data Refresh schedule.