LityxIQ has the ability to auto-load external data in batch. For example, it can be setup so that any file in a particular folder on an FTP site will be loaded once they are placed there, and continue to do so in an automated, ongoing manner. This document will describe the settings specific to setting up a batch loading process. Other settings related to the settings for a Raw Dataset are found here: https://support.lityxiq.com/261835-Define-the-Source-Settings-for-a-Raw-Dataset. This will help you create a Raw Dataset and open the settings window which will look like:
Step 1
After choosing your Location, select Template for Batch of Files from the Use Filename Template select box.
In LityxIQ, the imported datasets will be named using the name of the file that is imported. You can optionally have LityxIQ add a prefix to the front of each LityxIQ dataset name by entering it in the Prefix for Dataset Names box.
Click Folder to Find File and select the folder in which the files will be found.
In the Filename Template box, enter a string that represents the filename pattern for all files to import through this batch process. The asterisk (*) character can be used as a wildcard. For example, if you enter Myfiles*.txt, the batch loading process will search for any file in the selected folder whose name begins with "Myfiles" and ends with ".txt", and has any characters in between.
Other settings on this tab are as described in https://support.lityxiq.com/678242-Import-a-Delimited-Text-File.
Step 2
On the Dictionary tab (see https://support.lityxiq.com/261835-Define-the-Source-Settings-for-a-Raw-Dataset for details about the options on this tab), ensure that the Dictionary Load Options is set appropriately. The batch loader will automatically use this same option for all files it loads, so the choice you make depends heavily on the files and type of data you are loading, and the consistency from one file to another. Some recommendations are:
- If the files to be loaded are of very different structure, or their structure consistency cannot be guaranteed from one file or one load to the next, it is necessary to select the option Create Dynamically Each Import to avoid errors. This is the most common situation for a batch loading process.
- If all the files will be of the same structure each time and for every file, and you would like LityxIQ to determine that structure the first time a load occurs, select Create Automatically From Data.
- If all the files will be of the same structure each time and for every file, and you have a dictionary file prepared already, select Use Dictionary File.
Step 3
A batch loading process works automatically, so it needs to be setup on an Upon Data Refresh schedule. Please see https://support.lityxiq.com/373978-Execute-Raw-or-Derived-Datasets-on-a-Schedule for help setting this up.
Result
Within approximately five minutes, LityxIQ will begin searching the chosen folder and begin importing files into LityxIQ datasets. It will do this continuously until the batch loading dataset is taken off of the Upon Data Refresh schedule. In fact, if the same file appears in the selected folder again with a fresher date (for example, a refreshed version of the file), it will be detected and the associated LityxIQ dataset will be refreshed.
Note that each imported file will become a LityxIQ raw dataset, named as described above. You will need to refresh the Dataset list table (Actions button, select Refresh) to see the datasets as they make their way into LityxIQ. At this point, it is no different from any other LityxIQ dataset, and can be used as input into derived datasets or any form of analytics.