Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


HTML
<style>
.text-span-6 {
    background-image: linear-gradient(99deg, rgba(170, 163, 239, .5), rgba(125, 203, 207, .5));
    border-radius: 50px;
    padding-left: 15px;
    padding-right: 15px;
}

#title-text {
display: none;
}

.panelgradient {
    background-image: linear-gradient(180deg, #d5def0, whitesmoke);
    border-radius: 8px;
    flex-direction: column;
    justify-content: center;
    align-items: center;
    padding: 4rem;
    display: flex;
    position: relative;
}

</style>


<div class ="panelgradient">

<h1 style="text-align: center;">Datasets <br> (Databases and SQL Queries)</h1>

</div>



Introduction to the Datasets Module

Image Added

The Datasets Module is designed for data exchange with SQL databases and text files from a diverse set of sources. Essentially, the Datasets Module drives bi-directional real-time communication between all modules and the SQL databases.

This module offers compatibility with various database technologies, including ADO.NET, ODBC, OleDB, and native interfaces, providing straightforward configuration with prominent databases such as SQL Server, Oracle, SQLite, and PostgreSQL. Features include:

  • Multi-threaded concurrent connections with multiple databases for efficient data handling
  • SQL Query editor, SQLite admin tool, and a Visual Query Builder, streamlining the configuration experience
  • Customization of SQL statements in real-time with tags and system events

Image Added

On this page:

Table of Contents
maxLevel3
minLevel2
stylenone


Key Concepts and Terms

DatasetDB

Connections settings created by the Dataset Module to communicate with an external database.

DatasetQuery

Logical name associated with the configuration for SQL query statements with a Database, and its properties and methods for running que queries.  

DatasetTable

Logical name created to hold configuration settings to access specific tables in a connected database, mapping tags to table columns for operations.

DatasetFile

Logical name defining parameters for reading and writing files in ASCII, Unicode, or XML formats.


Understanding the Datasets Module

The Datasets Module enables users to interact with SQL databases seamlessly. The module supports real-time Tags within SQL statements, and manages files and recipes in ASCII, Unicode, or XML formats.

The data retrieved from databases can be utilized in various ways throughout your solution. For example:

  • In the Displays Module: Visualization tools like DataGrids can present query results on screens and dashboards, creating custom views of the data that are accessible and easy to understand for your users.
  • In the Scripting Module: Custom scripts can reference query results and trigger specific actions, such as sending notifications, updating tags, or performing calculations, thereby implementing complex logic based on database data.
  • Devices:  Sending data from field equipments to a SQL database, or applying settings from the database to the field equipments.

Pre-defined Database Connections

The Dataset Module also serves as a data storage configuration hub for other modules. The following Database connections are pre-defined by the Dataset Module.

  • AlarmHistorian: Events and records for long-term retention.
  • TagHistorian: Time-series storage for process variables,
  • RuntimeUsers: Dynamics users and credentials created when running the solution.
  • Retentive: Persistent records for tags and properties that need to be kept across multiple starts of the solution (typically configuration settings and setpoints).

Processing Data Requests

The Datasets Module has its implementation running as a service, which ensures high performance and real-time responses to multiple client requests.

This architecture also enhances protection and security for the database, as client displays and scripts won't access the databases directly, but through the Datasets Service.

Another benefit is the ability for Data Source Virtualization, meaning that when the solution is using Dataset.Query.Query1 in its displays or scripts, the database running that query, along with the query itself, can be maintained or replaced without affecting the overall solution configuration. This feature allows the solution to work with the data, regardless of the underlying data storage technology.

For an advanced deeper understanding of the Datasets Services, see Dataset Advanced Topics.


Configuring the Datasets Module

Configuration Workflow

The typical configuration workflow for the Dataset module has the following sequence:

Datasets Module Configuration Workflow

Action

Where 

Comments

Define database connections

Datasets→DBs

Gather connection details for your applications databases and created DB objects as need. Leverage the built-in SQLite admin tool for temporary development purposes

Prepare Queries

Datasets→Queries

DataExplorer→SQL

VisualQueryBuilder

Queries to craft queries using the built-in SQL Language Editor, the  VisualQueryBuilder or using provided SQL statements from other sources.

Fine-tune queries adding real-time parameters. Eg.: Transform "WHERE col1 = 5" to "WHERE col1 = {{tag.Test}}".

Map Database Tables

Datasets→Tables

Optionally, you can establish a direct mapping to tables within the Database. 

Map Recipes and Text files

Datasets→ Files

Optionally, your solution may need to save or load recipes, or other information, from ASCII, Unicode, or XML files. 

Managing DB Connections

There are four database connections pre-defined in any new solution.

Datasets DB - Pre-defined database connections

DB

Database

Path Location

Usage

Retentive

SQLite

<ProjectNameAndPath>.dbRetentive

Stores values for the Tags with the Retentive property set.

RuntimeUsers

SQLite

 <ProjectNameAndPath>.dbRuntimeUsers

Stores dynamically created Solution SecurityUsers.

AlarmHistorian

SQLite

 <ProjectNameAndPath>.dbAlarmHistorian

Stores Alarm and AuditTrail records.

TagHistorian

SQLite

<ProjectNameAndPath>.dbTagHistorian

Stores Tag Historian and Annotations.


When using SQLite databases, the Dataset Module can automatically create the database locally if it doesn't already exist. For other database types, the database itself must already exist before you set your connection.

→ Read more about Datasets DBs.

DatasetQueries Configuration

Use the DatasetQueries to define SQL Statements, for queries and stored procedures to execute in connection of the created DatasetDB databases. 

Read more about Datasets Queries

DatasetTables Configuration

Use the DatasetTables to access or exchange data with databases tables, with simplified query syntax. It allows allow insert new rows directly on database tables.

Read more about Datasets Tables.

DatasetFiles Configuration

The DatasetFiles are used to customize file interactions in the Dataset Module. With this feature you can read or write realtime tags to ASCII, Unicode and XAML files.

Read more about Datasets Files.


Working with the Datasets Module

Runtime Execution

When executing the solution, there is an infrastructure of services that manages access to the database and transports that information to where it is requested. For instance, to display a Dataset Query result on an HTML5 page, that request first goes to the server, which then requests the database (which can be on another computer), and the information flows back to the requester.

As database operations can take some time to execute, it is very important to understand some aspects of the Datasets Module execution, including the concept of synchronous vs. asynchronous requests.

The page Datasets Module Execution details concepts that describe the module's internal operations.

Showing DataGrids Tables on Displays

One typical use of the Dataset Module is to display query results on displays and dashboards.

In order to so, create or DatasetQuery, or DatasetTable, then use the DataGrid Control on your displays.

Using Query Results on Scripts and Tags

It's possible to define the SQL statements with code (either using the Scripts Module or Display CodeBehind) and connect the results with tags early on.

The property Dataset.Query.Query1.SqlStatament holds the query that will be executed; just modify that property within your scripts. 

The Tag Type DATATABLE was created to be compatible with results of Select() statements. SImply apply the results to your query and use tags no manage the information. 

The TK (Toolkit extension for Scripts) has methods that allow for easy copying between DataTables (query results) and Template Tags, like TK.CopyTagToDataTable().

Monitoring Databases Connection Status

Monitoring Database Connections is an essential aspect of maintaining a reliable operation of the solution. 

This can be accomplished using the Dataset Namespace properties, which provide status for DatasetTables and DatasetQueries operations.

Read more about Datasets Runtime Attributes.

During the Development phase, when the Designer tools is connected with a Runtime (the Solution is in execution), the main status conditions can be seen in the monitoring page.

Read more about Datasets Monitor


Datasets Advanced Topics

Datasets Module Execution

The Dataset module facilitates efficient database interactions by utilizing TServer services, managing synchronous and asynchronous executions for optimal performance.

Read more about Databases Connection And Interactions.

Data Management

The Dataset Module offers versatile methods for managing data and concurrency within solutions, including Data Table tags and Async Contents.

Read more about Data Management.

Datasets Runtime Attributes

The Datasets Namespace exposes properties and methods from the .NET objects used by the Historian Module execution. You can use these properties and methods on your Displays or to create Scripts and Alarms.

Read more about Datasets Runtime Attributes.

Preventing SQL Injections

See page Datasets Advanced Topics

Network Gateway Access And Time Zone Handling

See page Datasets Advanced Topics

Backup Of Solutions SQLite Data Bases

See page Datasets Advanced Topics


Anchor
BestPractices
BestPractices
Best Practices and Troubleshooting

Common Issues and Solutions

Connection loss between project and database

Database Timeout Configuration: The database may have a timeout setting that automatically disconnects idle connections after a certain period. It's recommended to check the database's timeout setting and adjust it, if necessary, to ensure that the connection remains active overnight.

Power Settings: It's also suggested to check the computer's power settings to ensure that it doesn't enter sleep or hibernation mode during idle moments, which could cause a loss of connection to the database. Adjusting these settings to keep the computer active during these idle moments may resolve the issue.

Database Connection Problem

In the DB configuration, there is always a "Test" button to ensure that the connection is happening correctly. When there is a problem, the return of this button is an error message, usually returned by the database provider itself. The most common errors are: invalid user, invalid password, computer without access to the database, incorrect form of authentication.

Issue: Error accessing the Database Table

Once the connection is established, the Table configuration is specific to a table. In the "Table" combobox, the list of available tables automatically appears. It is possible, via script, to change which table will be accessed. However, care must be taken that the table exists and that the configuration is done using the correct name. The same care must be taken when Queries are used, as it is the user's responsibility to type the correct table name, as well as the syntax of the separators.

Error in the Syntax of the Query

It is the user's responsibility to type the correct SQLStatement of a query using the QueryBuilder. Table name, column, values, all can generate an error if used incorrectly. For example: comparing different types may not return the expected result, strings in general should be in single quotes. The separators and clauses available can vary between databases. For example:

SQLServer

Code Block
languagesql
titleQuery syntax
SELECT TOP 10 * FROM table WHERE column = value

SQLite

Code Block
languagesql
titleQuery syntax
SELECT * FROM table WHERE column = value LIMIT 10;

Oracle

Code Block
languagesql
titleQuery syntax
SELECT * FROM table WHERE column = value AND ROWNUM <= 10;

or new Oracle version

Code Block
languagesql
titleQuery syntax
SELECT * FROM table WHERE column = value FETCH FIRST 10 ROWS ONLY;

IBM DB2

Code Block
languagesql
titleQuery syntax
SELECT * FROM table WHERE column = value FETCH FIRST 10 ROWS ONLY;


ServerIP without TWebServer Running on the Remote Machine

In some cases, the computer may not have access to the database. In this case, it is possible to create a gateway, routing the commands to be executed on the computer that has access to the database. The ServerIP field should be configured with the IP and port (<IP>:<port>), pointing to this computer that has access permission. This computer must have the software with the TWebServer running installed. It will automatically perform this gateway service and send the commands to the database.

DataTable Returned NULL

When a query is returned null, some error has occurred. Some common errors include: connection failure with the database, table not found, Dataset module is not running, incorrect query syntax. Check the return of the method using WithStatus when using a synchronous method or use the LastStatus and LastStatusMessage property when using asynchronous mode.

DataTable Returned with 0 Rows

When this happens, in general, there is a connection with the database and the table name is correct. In this case, the search condition is usually wrong, or the table is really empty. Check if the column names are correct and if the separators and clauses are valid.

Dataset Module is Down

Although the TServer is responsible for forwarding requests to the database, the management and communication with the TServer is done by the Dataset module, as well as the treatment of responses. Therefore, if you are having basic problems with access and execution of database access, the first thing to check is whether the module is set up to run and is actually running.

Very High Response Time

Sometimes, it may seem that access to the database is not being made, but what might actually be happening is that some accesses can return a very large amount of data, or the database may be overloaded, or with a bad configuration, causing it to have a low performance. Or even, the network itself can be overloaded and slow, and all these factors can impact the response time. In these cases, it is important to execute the Query directly in the database environment to be sure that the problem is not on the side of the database. Do this and check how long the database itself takes to execute the query. It is also worth checking the volume of data exchanged to have an idea of the related side effects.

Update of a table with the wrong schema (select before update)

The Dataset module uses ADO technology, and many things are resolved at the level of this API. When we are going to perform an Update on a table, the schema of the table and controls in the .NET DataTable type are used. Therefore, if you are going to perform an update passing a Tag or .NET DataTable object as a parameter, it is important that this object respects the schema of the destination Table in the database. Normally, a Select command must have been given at some point to obtain the correct schema used by the bank. After this, it is easy to add, remove, and modify values in this DataTable and update it back to the physical table in the database.

Where condition CaseSensitive

Case sensitivity in a WHERE clause depends on the database and the configuration of the database you are using. For example, in MySQL, queries are case-insensitive by default, which means 'abc' and 'ABC' would be considered equal. However, this can be changed with specific database settings. In SQL Server, case sensitivity is also determined by the database configuration. In PostgreSQL, queries are case-sensitive by default, so 'abc' and 'ABC' would be considered different. Therefore, it really depends on the specific database and the settings of that database. If you need to ensure case-insensitivity in a query, you can use functions like UPPER() or LOWER() to convert all values to upper or lower case before comparison. For example:

Code Block
languagesql
titleQuery syntax
SELECT * FROM table WHERE LOWER(column) = LOWER(value);

This query will return records where the column matches the value, regardless of capitalization.

Performance

The Dataset module's performance depends on many factors, including database performance, network latency, and the complexity of executing SQL queries. The platform will minimize overhead and execute queries as efficiently as possible. However, ultimately, the performance of the Dataset module is tied to these external factors. It's essential to design your database schema and queries with performance in mind and consider investing in high-quality hardware and network infrastructure to ensure optimal performance.

Best Practices and Recommendations

Error Handling

Error handling in the Dataset module is straightforward. If an error occurs during the execution of a command, the error message will update the module's Error property (Last Status). You can monitor this property to handle errors in your application. Furthermore, if an error occurs during the execution of a synchronous method, the process will return an empty DataTable and update the Error property. Alternatively, you can call methods like SelectCommandWithStatus, where the status will be an output parameter in the method.


In this section:

Introduction

The Dataset Module is a versatile tool specifically designed for data exchange with SQL databases and text files from a diverse set of sources within the platform. This module is responsible for shaping the foundational structure for data interaction, which in turn allows other real-time modules on the platform to function effectively. It is crafted to provide seamless compatibility with a wide range of database technologies, including ADO.NET, ODBC, OleDB, and native interfaces. This ensures straightforward configuration with essential databases such as SQL Server, Oracle, SQLite, and PostgreSQL.

Constructed with real-time applications at the forefront of its design, the Dataset Module incorporates several essential features. These include multi-threaded concurrent connections, a dedicated SQLite editor, and an intuitive visual query builder. This functionality not only simplifies query creation and manipulation but also streamlines the overall data exchange process. Additionally, the Dataset Module can integrate real-time tags within query strings and manage files and recipes in ASCII, Unicode, or XML formats. This versatility amplifies the module's overall functionality and enhances the user experience by providing a smooth, efficient interface for data exchange.

On this page:

Table of Contents
maxLevel3

Purpose and Key Concepts

Dataset DBs

In order for the Dataset Module to communicate with an external database, a connection must be created with certain parameters. These connections, which are created within the Datasets → DBs section of the module, are referred to as Dataset DBs.

Dataset Queries

In the context of the Dataset Module, a Dataset Query refers not only to an SQL query string, but also to a Project object that has a logical name, an SQL query related to that logical name, and other parameters defined within the Datasets → Queries section. There are many ways to automatically map the results of a query execution with Tags.

Dataset Tables

A Dataset Table is a logical name that is created within a project to set up access to a specific table in a connected database. These tables are listed within the Datasets → Tables section of the module. The Tags in the real-time database can easily be mapped to columns in the tables to perform insert, update, or read operations.

Dataset Files

A Dataset File is a logical name that defines parameters for reading and writing files in ASCII, Unicode, or XML formats.

Understanding the Datasets Module

Overview

The Datasets and SQL Queries module provides a powerful and flexible way to interact with databases, exchange data, and integrate this data into your projects. This module allows you to connect to various databases, execute SQL queries, and retrieve or manipulate data as required by your application.

Data exchange with the database is performed through a combination of queries and/or stored procedures. These can be executed on-demand, on a schedule, or triggered by events within your project. The module supports various database providers, including SQL Server, Oracle, and MySQL, enabling seamless integration with different database systems.

The data retrieved from the database can be utilized in various ways throughout your project. In the Displays module, you can use visualization tools like DataGrids to present query results on screens and dashboards. This allows you to create custom views of the data, making it accessible and easy to understand for your users.

In the Scripting module, you can create custom scripts that reference query results and use them to trigger specific actions, such as sending notifications, updating tags, or performing calculations. This enables you to implement complex logic based on the data from your database, making your project more dynamic and responsive to changes in the underlying data.

You can use the data retrieved from databases in the Datasets module for example to configure or update alarm conditions, triggers, and actions. This enables you to create dynamic alarm systems that respond to changes in your data, ensuring that your application stays responsive to real-world conditions.

Features Highlights

  1. SQL Query Support: The Dataset Module provides support for SQL queries, allowing users to easily extract, manipulate, and transform data from a variety of sources.

  2. Integration with External Data Sources: The Dataset Module can integrate with a wide range of external data sources, including databases, CSV files, and other external sources, providing a flexible and powerful tool for data collection and analysis.

  3. Access Types: Access Types allow users to group and organize data points based on their usage and permissions, providing a powerful tool for managing and controlling access to data within the Dataset Module.

  4. Visual Query Editor: The Visual Query Editor provides a user-friendly interface for creating and editing SQL queries, making it easy for users to define complex queries without needing extensive SQL knowledge.

  5. Customizable Dashboards: The Dataset Module provides the ability to create custom dashboards and visualizations based on the data collected by the module, allowing users to easily view and analyze data in real-time.

  6. Store Procedures Execution: The Dataset Module can execute Store Procedures and return the results to the platform, allowing users to perform advanced data manipulation and analysis within the context of the platform.

  7. Real-Time Execution: The Dataset Module supports real-time execution of SQL queries, allowing users to monitor and analyze data as it is generated in real-time.

Datasets Data Server Service

The Datasets Data Server Service is an essential part of the Datasets Module. It is responsible for managing and providing efficient access to datasets, which are structured collections of data used in various project applications such as data analysis, reporting, and visualization. The service ensures high performance and seamless integration with other components, offering flexibility and ease of use when working with datasets.

Processing Requests from Other Modules

The Processing Requests from Other Modules feature in FactoryStudio enables smooth communication between different project modules. It handles data requests and interactions, ensuring efficient data exchange and coordination among various components such as HMI/SCADA screens, scripting, and reporting tools.

Databases Used when Running the Project

The platform has pre-configured databases that store essential project information, including real-time and historical data, alarms, events, and system configurations. These databases provide a reliable and efficient foundation for data storage and retrieval, allowing users to focus on building and customizing their projects without worrying about database setup and management.

Data Source Virtualization Benefits

Data Source Virtualization is an advanced feature that simplifies data management across multiple data sources. It offers a unified interface for accessing, querying, and manipulating data, regardless of the underlying data storage technology, ensuring flexibility and ease of use.

Agnostics, Standards, Centralized Management

FactoryStudio's Data Source Virtualization is designed to be agnostic and adhere to industry standards, allowing it to work seamlessly with various data storage technologies, such as SQL databases, OPC UA servers, or custom data sources. This approach enables centralized management of data connections and configurations, streamlining the process of integrating different data sources into your project.

Detailed and Advanced information

The execution of the Dataset module, i.e., executions of configurations made in Queries, Tables, and Files, runs in an independent process "TRunModule.exe (Dataset)" that reads all the project configurations related to this module but does not directly execute the interface with the databases. Within TServer, there are services to access the database, and connections and calls to the database occur in TServer. The Dataset module merely consumes this bank access service from TServer.

  • Connections with the Database
    For each configured DB, there's only a single access connection created for this database, meaning that there's only one execution thread for each DB. If parallel execution of commands to the same database is needed, it's possible to create more than one DB pointing to the same database.

  • Synchronous and Asynchronous Methods
    Both in Tables and in Queries, if the properties are used to trigger the execution, such as triggering the "Select" property, this will make the execution Asynchronous. In other words, after the trigger in the property, the execution will continue normally, and in parallel, this property will propagate from where it suffered the trigger (screen or script) to the server, then being synchronized with the Dataset module which in turn will make the necessary calls to the TServer's bank access service. After the return of the TServer with the execution of the Select for the Dataset module, the result will be updated in the properties and attributes of the module, the mapping treatment with tags will be done, and the result will be available to the entire project.
    As for the available methods, the execution will be synchronous. For example, when calling the SelectCommand method, the execution will be locked in this method while the Dataset module executes the call of the TServer service, which interfaces with the database returning the result to the Dataset module. The Dataset module then processes the mapping with tags and only then provides the result in the form of the method's return, allowing the continuation of the execution.
    Therefore, it's important to be careful with synchronous calls within code-behind screens, which are mono-thread, whether they are WPF or HTML5 screens. Mechanisms like using Script task or Action Server Request can help make an asynchronous execution without needing to use the attributes, when there's a need to handle the result even after an asynchronous execution.
  • Tags Domain Client Used in Mapping
    It's possible to use client-type tags in the Dataset module's tag mapping. When executing a synchronous or asynchronous command, the assignment in the tags will be executed by the Dataset module (not by TServer) in the original domain of the call. Therefore, if the call was executed on a client (screen or client script), after the return of the result from TServer to the Dataset module, the mapping will be executed between received values and tags, and Client domain Tags can be used in the mapping. It's important to understand and consider if the project's need is an execution that makes the value available to the entire project, in this case using server domain tags, or only for the client's scope, in this case using client domain tags.

  • Attributes and Server Domain Properties (concurrence management)
    It's important to know and remember that the module's attributes are all server domain. Therefore, if there are multiple Tasks or clients using the same attributes, there may be competition and confusion in execution. For instance, a client sets the SQLStatement of a query, and at the same time, a Task also sets this SQLStatement with a different value from the client. During the command execution (for example, a select – triggering the Select property which is also a server domain), the last value in the SQLStatement is what will be used in the operation.

  • Use of Tag DataTable
    The return of the Select methods is a .Net DataTable object. If you're not using mapping to assign each value from a specific row's columns to Tags, you can manipulate this returned dataTable directly in the script, or you can assign it to a Tag of the DataTable type. This allows the result to propagate to other modules, like clients or reports, and be easily presented in a dataGrid, PieChart, or even for XY chart presentation using the CallBack GetSamples.

  • DataTable Properties Available in the Dataset Module
    As mentioned, there are several server domain properties within the Dataset module. There is a property called AsyncContents, which is a .Net DataTable type. When the Select property is triggered, initiating an asynchronous execution, after the return, the values of each column of the 1st row are assigned to the tags that were configured in the Mapping. When the Next property is triggered, the Dataset module navigates to the next row and assigns the column values to the tags, even those configured in the Mapping. The table used for this navigation is stored in the AsyncContents property and is updated only when the select is executed. It's possible to modify the CursosIndex to jump or go back to a specific position in the row navigation.
    For synchronous commands, the method returns the dataTable, but it's also stored in the LocalContents property; it will be overwritten with the result of each executed command.

  • Store&Forward
    Only banks used to store Historian and Alarm can have the Store&Forward functionality enabled because this feature requires the use of some control columns that exist in the schema of Historian and Alarm, and not in generic banks. Please refer to the Store&Forward section of the Historian and Alarm modules for more details.

  • Data Traffic Volume Between TServer, Dataset Module, and Other Modules
    When a select is executed in the bank, the WhereCondition property of the Dataset>Table defines the return condition just like the SQLStatement property of the Dataset>Query. Therefore, the volume of data that can be returned can vary from no row to the entire table (when there's no filter or a WhereCondition). There is no software-side protection concerning the size of the return DataTable, and obviously, the larger the table size, the more allocated memory, the longer the traffic time of this table between TServer and the Dataset module, and then its final destination. Care should be taken with the scalability of these tables' size, as well as having suitable hardware to support the expected data volume.

  • Specific Syntax Used in the SQLStatement of Database
    The SQLStatement property of Dataset>Queries allows for the full SQL statement to be executed, therefore it's possible to use SQL syntax specific to each bank. For example, SQLite supports the LIMIT clause, which in SQLServer is called TOP. Just as care must be taken with the separators of table and column names, some use [] others "". As much as possible, the software will try to normalize and use the correct separators when using the WhereCondition and filters on objects with the alarm object, but in SQLStatement, it's up to the user to use the correct syntax.

  • Protection Against SQL Injection
    In the database, calling Stored Procedure, there is a great concern about this part of "injection," because if parameters are passed as plain text in SQL Statement, the "injection" could be possible. Against this, we use the .Net API where parameters are added to a list, making it impossible to code injection.
    Therefore, it's advisable to use parameters in the Stored Procedures calls following the following syntax:
    "execute <StoredProcedureName> <@paramName1>={Tag.Tag1}, <@paramName2>="{Tag.TagStr}", <@paramName3>={Tag.OutValue} OUTPUT, ... , @RETURN_VALUE={Tag.RetValue} RETURNVALUE"
    By using this syntax, you effectively parameterize the Stored Procedure calls and prevent SQL injection attacks. This way, each parameter value is interpreted strictly as a value and not part of the SQL command. This practice is fundamental to the security of your application, preventing unauthorized manipulation of the database.
  • Using ServerIP to Access Remote Database Providers
    There are times when certain databases are not accessible from any computer. By using the ServerIP column, it's possible to redirect database access commands to a computer that has the required permissions. The prerequisite for this is that the software should also be installed on that computer. In this scenario, only TWebServer would be running. This system is responsible for interfacing with the database, receiving commands, establishing connections, etc. This method provides a workaround for situations where database access is restricted to specific machines.
  • Time Zones
    All Date/Time type tags in the software are treated as UTC. Therefore, the dates that are written and read in the database are also in UTC. If you want to read/write a date in a local timezone, you should consider the time difference when preparing the SQL statement or when using the WhereCondition property.

  • Dataset Module Performance
    The Dataset module's performance depends on many factors, including database performance, network latency, and the complexity of the SQL queries being executed. The software itself is designed to minimize overhead and execute queries as efficiently as possible, but ultimately the performance of the Dataset module is tied to these external factors. It's important to design your database schema and queries with performance in mind, and consider investing in high-quality hardware and network infrastructure to ensure optimal performance.

  • Error Handling
    Error handling in the Dataset module is straightforward. If an error occurs during the execution of a command, the module's Error property (LastStatus) will be updated with the error message. This property can be monitored to handle errors in your application. Furthermore, if an error occurs during the execution of a synchronous method, the method will return an empty DataTable and the Error property will be updated or you can call methods like SelectCommandWithStatus where the status will be an output parameter in the method.

  • These are some of the key points you should keep in mind while using the Dataset module. It's a powerful tool that provides a lot of flexibility, but it also requires careful use and understanding of its features and limitations. Always keep best practices in mind when designing and implementing your database interactions, and don't hesitate to seek expert advice if you're unsure about anything.

    Configuring the Dataset Module

    Learn how to connect to data sources, create queries, and optimize performance for efficient data management.

    This section provides essential guidance for setting up and customizing the Dataset Module, including:

    AnchorWorkflowWorkflowConfiguration Workflow

    The typical configuration workflow for the Dataset Module has the following sequence:

    Create the required database connections (DBs)

    1. Go to Datasets → DBs
    2. Collect the information to connect with the databases required for your project.
    3. Use the built-in SQLite database as a temporary development tool if one of your connected databases is not available yet.
    4. The virtualization model with logical names for queries and tables will make your project work directly with the new connection with the production database, without having to change anything on the Project Configuration other than that database connection.

    Prepare the Queries the Project uses

    1. Go to Datasets → Queries
    2. Either use the Visual Query Editor or get the query string from IT or plant facilitator.
    3. Collect and create the logical names Dataset.Query to identify those queries.

    Modify the Query to add real-time tags

    1. Go to Datasets → Queries
    2. Easily modify the query with the parameters that need to be connected with real-time values.
    3. For instance, a query that has the text "WHERE col1 = 5" can be modified to "WHERE col1 = {{tag.Test}}". The value of the Tag will be added to the proper position when the query is executed.

    Prepare the Tables the Project uses

    1. Go to Datasets → Tables
    2. When you need to Insert or Modify data, you need to access the Database Table directly.
    3. In some cases, all the information you need is in one table, so there is no need to create a Query.
    4. You can easily connect the contents that are inserted in the table with Tags in the Project.

    Configure the Stored Procedures

    1. Go to Datasets → Queries
    2. The Module Database can execute Stored Procedures; just define it using the same interface for the queries.

    Configure data exchange with Files

    1. Go to Datasets → Files
    2. If necessary to exchange values of Tags with plain text or XML files, set that configuration.

    Use your Dataset logical objects

    1. The logical object names created for Queries, Tables, and Files can be used in any part of the project.
    2. Examples: Script calculation, Display visualization, and others.
    AnchorConnectionsConnectionsManaging DB Connections

    When using SQLite databases, the Module Dataset can automatically create the Database if necessary; for other ones, the Database itself must already exist before you set your connection. Users with any Permission groups can create new connections in the Project, but only the Administrator can configure databases password logins.

    To create a new Database connection:

    1. Go to DatasetsDBs.

    2. Click Create New. The Create New Database Connection window displays.

    3. Enter or select information, as needed.

    4. Click OK. The database is added as a new row in the table.

    5. Edit the row fields to modify the required settings.

    Dataset DB Configuration Properties 

    Column

    Description

    Name

    Enter a name for the database configuration. The system allows you to know if the name is not valid.

    Provider

    Identifies the Provider technology used in this connection 

    Database

    Identifies to which type of dataset is this connection

    ConnectionString

    Enter the information needed to connect with the database. The syntax is defined by the Database Provider used, it is a third party component, any additional parameter supported by the provider can be used normally.   

    You use macros on the connection string too.  

    Example: for the filename in a SQLite connection string, use <ProjectName> that is replaced by the name of the project.

    LogonName

    Enter a valid login name for the database.

    LogonPassword

    Enter the password that corresponds to the database login. (Only accessible by Administrators)

    ServerIP

    Optionally, an IP or DNS name for a computer to be used as a Secure Gateway.

    Description

    Enter a description for the database connection.

    Please check the Connecting to SQL Server and Connecting to Excel for additional information.

    There are four database connection already created in any new Project:

    Datasets DB - Pre-defined database connections

    DB

    Database

    Path Location

    Usage

    Retentive

    SQLite

    <ProjectNameAndPath>.dbRetentive

    Stores values for Retentive Tags.

    RuntimeUsers

    SQLite

     <ProjectNameAndPath>.dbRuntimeUsers

    Stores dynamically created Users.

    AlarmHistorian

    SQLite

     <ProjectNameAndPath>.dbAlarmHistorian

    Stores Alarm and AuditTrail records.

    TagHistorian

    SQLite

    <ProjectNameAndPath>.dbTagHistorian

    Stores Tag Historian and Annotations.

    Any of them can be customized to any type of database. 

    The selection of best storage location depends on all kind of factors, from internal company procedures to the volume of data and how the data shall be used. Therefore, that is decision to each Project according to its requirements.

    If needed to use another database for the pre-defined connections, execute the following steps:

    1. Rename or Delete the previous DB. This step is necessary, as the system would not allow to create two objects with the same name. 

    2. Crate a new DB with the same name of the previous DB, with the required Database and connection strings.

    ConnectionString example for SQL Express 

    Image Removed

    • Data Source: The server path and instance that will have the databases.
    • Initial Catalog: The name of the database that will be used.

    Project Test Databases and Example

    Assume you need to transition the AlarmHistorian configuration to a Microsoft SQL production database within your organization.

    During the development and testing phases of the application, you may prefer not to publish alarm events to that database yet.

    In other platforms, it would be necessary to manually switch connections between test and production environments or devise workarounds to handle this situation.

    In our framework, we provide a built-in, optional feature: configure the AlarmHistorian DB to target the production database, regardless of its current availability or intended use. Subsequently, run the project in Test Mode, storing data in the local SQLite file <projectName>.dbAlarmHistorianTest.

    When running the Project in Validation Mode, there is a configuration, which is true by default, that can override the connection of the pre-defined DB, using testing ones instead.

    Those are database files that can be enabled to use when running in Validation Mode:

    Database files used when running in Validation Mode

    DB

    Database

    Path Location

    Usage

    Retentive

    SQLite

    <ProjectNameAndPath>.dbRetentiveTest

    Stores values for Retentive Tags.

    RuntimeUsers

    SQLite

     <ProjectNameAndPath>.dbRuntimeUsersTest

    Stores dynamically created Users.

    AlarmHistorian

    SQLite

     <ProjectNameAndPath>.dbAlarmHistorianTest

    Stores Alarm and AuditTrail records.

    TagHistorian

    SQLite

    <ProjectNameAndPath>.dbTagHistorianTest

    Stores Tag Historian and Annotations.

    AnchorDatasets Queries ConfigurationDatasets Queries ConfigurationDataset Queries Configuration

    You can configure queries to perform more advanced functions with SQL statements to work with data from external databases.

    To configure Dataset queries:

    1. Go to DatasetsQueries.
    2. Enter the field values as needed.

    Dataset Query Configuration Properties  

    Column

    Description

    Name

    Enter a name for the query. The system allows you to know if the name is not valid.

    DB

    Select the database configuration.

    SqlStatement

    Enter the query using SQL syntax.

    Mapping

    Click "..." to select the tags that you want to populate with data from specific columns returned by the query.

    MappingDateTime

    Select the time reference (UTC or Local).

    Description

    Enter a description for the table configuration.

    With the Visual Query Editor, users can drag and drop tables, define relationships, and add filters and conditions using a simple graphical interface. Once the query is created, it can be saved and executed like any other query within the Dataset Module.

    Check the Visual SQL Query Builder page for complete information.

    AnchorDatasets Tables ConfigurationDatasets Tables ConfigurationDataset Tables Configuration

    To configure dataset tables:

    1. Go to DatasetsTables.
    2. Enter the field values as needed.

    Dataset Table Configuration Properties  

    Field / Column

    Description

    Name

    Enter a name for the table configuration. The system lets you know if the name is not valid.

    DB

    Select the database connecton.

    TableName

    Select or type the table name in the Database you want to access

    WhereCondition

    Specify the parameters that will filter the data using SQL syntax. E.g. "ColumnName = {tag.tagInt}"

    Access

    Select the access permissions for the table.

    Mapping

    Click "..." to select the tags that you want to populate with data in the first row of the table with data from specific columns.

    MappingDateTime

    Select the time reference (UTC or Local).

    Description

    Enter a description for the table configuration.

    AnchorDatasets Files ConfigurationDatasets Files ConfigurationDataset Files Configuration

    To configure dataset files:

    1. Go to DatasetsFiles.

    2. Enter the field values as needed.

    Dataset File Configuration Properties

    Column

    Description

    Name

    Enter a name for the file configuration. The system allows you to know if the name is not valid.

    FileName

    Enter the full path to the file. The file path can have Tag values embedded using curly brackets syntax. E.g.: ExampleFile{{tag.Test}}.txt. 

    When executing, the area in curly brackets is replaced by the value of the Tag.

    FileType

    Select the type of file.

    Objects

    Click "..." to select the tags that you want to populate with data from the file with data from specific columns.

    Description

    Enter a description for the file configuration.

    XMLSchemaType

    Represents the schema type of an XML file, which can be: a TagList, XML that contains a tag list with the tag name and tag value; or a TagObject, XML that contains the entire tag tree and its children.

    Working with the Dataset Module

    Runtime Execution

    One of the key features of the Dataset Module is the ability to execute SQL queries and retrieve data in real-time. Here are some ways to leverage the runtime execution features of the Dataset Module:

    • Create SQL queries to retrieve data from external databases.
    • Use query results to trigger events and actions within the platform environment.
    • Configure event triggers based on specific query criteria, such as changes to a specific data point or a threshold value being exceeded.

    The Dataset Module can be easily integrated with other modules within the software environment. Here are some examples of how the Dataset Module can be used in conjunction with other modules:

    • Alarm Manager: Configure alarms based on query results to trigger notifications and actions.
    • Visualization: Display query results on screens and dashboards using DataGrids and other visualization tools.
    • Scripting: Use query results to trigger custom scripts and perform complex data processing and analysis.

    By leveraging these integration options, users can gain greater insight and control over their data sources within the platform. With the ability to execute SQL queries and trigger actions based on query results, the Dataset Module provides a powerful set of tools for working with data.

    Monitoring Databases Connections

    Monitoring Database Connections is an essential aspect of maintaining a reliable and efficient system within the platform. By keeping track of database connections, you can ensure that your data is being accessed and updated correctly. Here are some ways to monitor database connections:

    1. IsStarted: This property indicates if the dataset is started, meaning it has been initialized and connected to the database. You can use it to check if the dataset is currently running. 

    Code Block
    languagec#
    titleExample
    bool isStarted = @Dataset.DB.YourDatabaseName.IsStarted;

    2. Query Execution Time: You can measure the execution time of a query by checking the time before and after executing the query. 

    Code Block
    languagec#
    titleExample
    DateTime startTime = DateTime.Now;
    @Dataset.Query.YourQueryName.SelectCommand();
    DateTime endTime = DateTime.Now;
    TimeSpan executionTime = endTime - startTime;

    3. ConnectionString: You can check the connection string used for the database to ensure it is configured correctly.

    Code Block
    languagec#
    titleExample
    string connectionString = @Dataset.DB.YourDatabaseName.ConnectionString;
    Note

    Please note that these are just examples, and you will need to replace YourDatabaseName and YourQueryName with the actual names of the database and query you are working with in the platform.

    Showing DataGrids Tables on Displays

    One of the key features of the Dataset Module is the ability to display query results on screens and dashboards using visualization tools like DataGrids. Here are some steps for using DataGrids to display query results:

    1. Create a query in the Dataset Module to retrieve the desired data.
    2. In the Displays Module, add a DataGrid control to the screen or dashboard.
    3. Configure the DataGrid to display the query results by selecting the data source and column mappings.
    4. Save and preview the screen or dashboard to display the query results on the DataGrid.

    Using Query Results on Scripts and Tags

    Users can use query results to trigger actions in custom scripts and tags. Here are some steps for using query results in scripts and tags:

    1. Create a query in the Dataset Module to retrieve the desired data.
    2. In the Scripting Module, create a custom script that references the query results.
    3. Use the query results to trigger specific actions within the script, such as sending notifications or updating tags.
    4. Save and execute the script to perform the desired actions.

    Check the Using Stored Procedures page for additional information.

    Troubleshooting and Best Practices

    Table of Contents
    maxLevel4
    minLevel3
    include#

    Common #Issues and Solutions

    Best Practices and #Recommendations

    Dataset Runtime Attributes

    The Dataset namespace exposes properties and methods of the .NET objects used by the Dataset Module execution.

    For more information on namespaces and objects, go to Objects and Attributes.

    Dataset Module Properties examples

    Property

    Type

    Description

    Example

    CursorIndex

    Integer

    Represents the current position of a cursor within the result set of a query. It is an integer value that indicates the row number in the result set that the cursor points to, with the first row having an index of 0. The cursor can be used to navigate through the result set, allowing you to read, update, or delete rows of data.

    Dataset.Query.QueryName.CursorIndex + 1

    OpenStatusMessage

    String

    Provides a human-readable message describing the current state of the database connection. This can be useful for monitoring the connection status, diagnosing issues, or displaying the connection status to users in a user-friendly format.

    Dataset.DB.DatabaseName.OpenStatusMessage

    ConnectionString

    String

    Used to store the necessary information for establishing a connection to a database. It is a string value containing details such as server name, database name, authentication credentials, and other relevant parameters.

    Dataset.DB.ProviderName.ConnectionString

    This section describes only some commonly used properties, for the full list properties and methods, go to the Namespaces Reference.

    In this section...

    Page Tree
    root@self
    spacesV10

    ...