NextTypes System

Third Party Software

The reference implementation uses Java, PostgreSQL and Tomcat. It is distributed in an easily installable WAR file and includes third party libraries to implement some components. In turn, these libraries depend on other libraries for their operation. It is convenient to know its use and license to develop applications on NextTypes.

Data Types

Each software uses different types of data, so it is necessary to define standard primitive and composite data types for use by all parts of the system. Data types are called "type" and are the core of the system and what gives the project its name.

Primitive Types

The main systems to integrate are PostgreSQL, Java and HTML5. It is necessary to make a comparison of the primitive data types of these systems to find what they have in common and the data types that are more or less standard in all computer systems although they receive different names in one system or another. In addition NextTypes adds other Java classes and data types for your use.

Characteristics of Primitive Types

Once the primitive data types have been identified, it is necessary to define valid characteristics and limits between the different systems to be integrated. All data types can also have the value null for integration with SQL.

tag1:attribute1,attribute2;tag2;tag3:attribute1

#30c74d

$2y$12$TFhXz5DvNAl0/jUg7hd56eLT52bZXiuBsDrapQ1Tm8LDaB7qaO3o6

Composite Types

Composite types are made up of a series of primitive types and links to other composite types. These links form a network database but using the properties of the relational databases. This system allows easy integration with programming languages and at the same time use all the capabilities of SQL. In the web interface you can see the example type "example", composed with all the primitive types and the composite type "article".

Composite data types have the following properties:

The characters allowed in type, fields, indexes and actions names are limited to numbers, lowercase ASCII letters and the character "_", with a maximum length of 30 characters, to ensure compatibility between different systems and limit the risk of code injection.

The properties of the fields are as follows:

The properties of the indexes are as follows

The definition of a composite type can be represented in JSON format or in another format for display, modification, storage or transfer to another system. Below is an example of the definition of the "article_language" type in JSON:

{
  "name" : "article_language",
  "cdate" : "2015-04-01T14:30:16Z",
  "adate" : "2018-09-26T14:59:35.53Z",
  "fields" : {
    "title" : {
      "type" : "string",
      "length" : 254,
      "not_null" : true
    },
    "language" : {
      "type" : "language",
      "length" : 100,
      "not_null" : true
    },
    "text" : {
      "type" : "html",
      "not_null" : true
    },
    "article" : {
      "type" : "article",
      "length" : 100,
      "not_null" : true
    }
  },
  "indexes" : {
    "al_ft_search_index" : {
      "mode" : "fulltext",
      "fields" : [ "title", "text" ]
    }
  },
  "actions" : { }
}

Objects

Similar to object oriented programming languages, an instance of a type is called an "object". Objects have the following properties:

article.id_name=select a.id, al.title as name from article a left join article_language al on (a.id=al.article and al.language=?)

When an object is stored in a system the storage format is the one used by that system. When that object is to be transmitted from one system to another it must be encoded in JSON or another format. In objects and fields of composite types the "name" can be included to expand the information but only the "id" is necessary. Following is an example of an "article_language" object in JSON:

{
  "type" : "article_language",
  "id" : "6539cb3a-cf6d-11e4-a8be-001f3c35b55c",
  "name" : "6539cb3a-cf6d-11e4-a8be-001f3c35b55c",
  "cdate" : "2015-04-01T14:30:16Z",
  "udate" : "2021-01-13T20:30:31.886528Z",
  "backup" : false,
  "fields" : {
    "title" : "NextTypes System",
    "language" : {
      "id" : "en",
      "name" : "English"
    },
    "text" : "NextTypes is a standards based system of storage, processing and transmission of information ...",
    "article" : {
      "id" : "system",
      "name" : "NextTypes System"
    }
  }
}

Universal Object Identifier

Each object is universally identified by its id, type and the server it is on.

host.example.com/type/id

This identifier can be used in QR codes or otherwise to access an object from any device or to tag objects real.

Export of Types and Objects

Types and objects are exported in two different types of containers. The "name" parameter of the objects is not included.

Object Container

The object container stores a group of objects of the same type. Its properties are as follows:

Here is an example omitting the objects:

{
  "format" : "nexttypes_objects",
  "version" : "1.0",
  "type" : "article_language",
  "count" : 6,
  "items" : [ ]
}

Types Container

The type container stores definitions of types and can also include containers for objects of those types. Its properties are as follows:

Below is shown an example omitting the types definition and the objects containers:

{
  "format" : "nexttypes_types",
  "version" : "1.0",
  "date" : "2020-11-14T22:53:00.688057Z",
  "types" : {
    "article" : {},
    "article_author" : {}
  },
  "objects" : {
    "article" : {},
    "article_author" : {}
  }
}

Systems Integration

Integration consists of adapting the NextTypes data types to the system to be integrated. Currently the main integrated systems are PostgreSQL, Java and HTML5.

PostgreSQL

This relational database manager with its large number of primitive types, the possibility of creating new types and its capabilities of Transactional DDL and deferrable constraints allows to create a data system with a high level of abstraction.

As indicated in the section "Primitive Types" several NextTypes data types correspond to PostgreSQL primitive types, some are specialized variants of these and some are new primitive types.

Each of the composite data types in NextTypes corresponds to a PostgresSQL table. Below you can compare the definition of the "article_language" type in JSON and the definition of the table with the same name in PostgreSQL using the "psql" program.

{
  "name" : "article_language",
  "cdate" : "2015-04-01T14:30:16Z",
  "adate" : "2018-09-26T14:59:35.53Z",
  "fields" : {
    "title" : {
      "type" : "string",
      "length" : 254,
      "not_null" : true
    },
    "language" : {
      "type" : "language",
      "length" : 100,
      "not_null" : true
    },
    "text" : {
      "type" : "html",
      "not_null" : true
    },
    "article" : {
      "type" : "article",
      "length" : 100,
      "not_null" : true
    }
  },
  "indexes" : {
    "al_ft_search_index" : {
      "mode" : "fulltext",
      "fields" : [ "title", "text" ]
    }
  },
  "actions" : { }
}

nexttypes=# \d+ article_language
                                               Tabla «public.article_language»
 Columna  |            Tipo             | Ordenamiento | Nulable  | Por omisión | Almacenamiento | Estadísticas | Descripción 
----------+-----------------------------+--------------+----------+-------------+----------------+--------------+-------------
 id       | character varying(100)      |              | not null |             | extended       |              | 
 cdate    | timestamp without time zone |              | not null |             | plain          |              | 
 udate    | timestamp without time zone |              | not null |             | plain          |              | 
 backup   | boolean                     |              | not null |             | plain          |              | 
 title    | character varying(254)      |              | not null |             | extended       |              | string
 language | character varying(100)      |              | not null |             | extended       |              | language
 text     | text                        |              | not null |             | extended       |              | html
 article  | character varying(100)      |              | not null |             | extended       |              | article
Índices:
    "article_language_pkey" PRIMARY KEY, btree (id)
    "al_ft_search_index" gin (to_tsvector('simple'::regconfig, (title::text || ' '::text) || text))
Restricciones de llave foránea:
    "article_language_article" FOREIGN KEY (article) REFERENCES article(id) ON UPDATE CASCADE DEFERRABLE
    "article_language_language" FOREIGN KEY (language) REFERENCES language(id) ON UPDATE CASCADE DEFERRABLE

The "id" identifier is a variable length text column of a maximum of 100 characters and is the primary key of the table. The columns description is used to indicate their NextTypes data type. The index with "fulltext" mode "al_ft_search_index" is an index that uses the PostgreSQL full text search system and the "language" and "article" composite types fields point to the "id" column of the "language" and "article" tables using deferrable foreign keys.

You can also inspect the data type with the pgAdmin graphical program or another graphical database management application.

Since the types do not usually change and their information is continuously required as it is the basis of the system, the description of the fields, indexes and content type of the fields are stored in memory so as not to have to continuously read this information from the database data and speed up system operation.

Java

In programming languages the types, fields, indexes, range of allowed values and objects are represented by the Type, TypeField, TypeIndex, FieldRange and NXObject classes. The Tuple class provides a common interface for accessing different resources such as an object, a row of an SQL query or the parameters of an HTTP request. Some classes are also added for HTML, JSON, XML, URL, Color, File, Image, Audio, Video and Document primitive types.

Unlike ORM or DSL systems, NextTypes does not define a correspondence between database tables and programming language classes but uses these generic classes to bring the relational model to the programming language.

HTML5

NextTypes makes use of the HTML5 forms with specific input elements for different data types such as "text", "tel", "url", "email", "password", "date", "time", "datetime-local", "number", "color", "checkbox" and "file". It also expands the controls of these elements with new graphic components and the use of text editors such as Codemirror and TinyMCE. It also makes use of the output elements like "a", "time", "img", "audio" and "video".

In the web interface you can see the "example" type with all the data types, their input elements and text editors.

System Types

The system depends for its operation on a minimum group of types that are listed below:

NextTypes System

Based on the standard data types for integration a system is created based on the concept of Model View Controller composed of the following modules:

• Node: Subclasses of the Node class that provides a data store with the basic actions of creating, reading, modifying and deleting types and objects.
• Controller: Subclasses of the Controller class that process information before reaching Node for the indicated types and allow additional actions to be taken.
• View: Subclasses of the View class. Views to create, view, and edit types and objects. Possible formats can be HTML, JSON, XML, RSS, WebDAV or iCalendar.
• Protocol: Modules that allow the execution of the Node, Controller and View methods through protocols such as HTTP or SMTP.

Configuration

The system has four types of configuration parameters:

• Types parameteres.
• Groups parameteres.
• Languages parameteres.
• Permisos.
• Modules parameters.

The configuration parameters are stored in ".properties" files that are read by the Settings class and its subclasses. The system includes files with default settings in "com.nexttypes.settings.defaults".

In the file WEB-INF/web.xml is defined the variable "settings_directory" that points to a directory where you can create a replica of those ".properties" files in which to give different values to the parameters. By default the directory is "/etc/nexttypes/settings" but it can be changed to another. If we have more than one NextTypes system on the same server, it will be necessary to create different directories for each system. Included in the source code are two sample configuration directories.

There is also the ability to modify the settings in the WAR file to distribute it with the desired settings.

Types Parameters

Each of the types can have different configuration parameters that are saved in the "types.properties" file and read with the class TypeSettings. The parameters are defined on a line in the following format:

type.property.subproperty.subproperty...=value

The "*" character can be used to assign the parameter to different types with the same root and their parts separated by the "_" character. For example, the "user*" parameters apply to "user" and "user_certificate" but would not apply to "users". Parameters assigned to "user_certificate" take precedence over "users*".

There are four types of configuration parameters:

• Simple: A single number or text string.
• Array: Multiple values separated by commas.
• Associative array: The ":" character separates the key from the value and commas the different elements.
• SQL: SQL query.

Groups Parameteres

The groups configuration files allow you to assign different values to the configuration parameters of the types for certain groups.

Language Parameters

These parameters are stored in a different file for each system language, they are read with the LanguageSettings class and are used for settings that differ between languages as text strings or date formats.

Permissions

Users are represented by the "user" type, groups by the "group" type and the membership of a user to a group by "group_user" type. The "guest" user represents unauthenticated users. By default the "administrators" group has permissions to perform all actions.

{
  "name" : "user",
  "cdate" : "2015-06-24T17:40:03Z",
  "adate" : "2015-06-24T17:40:03Z",
  "fields" : {
    "first_name" : {
      "type" : "string",
      "length" : 250,
      "not_null" : true
    },
    "second_name" : {
      "type" : "string",
      "length" : 250,
      "not_null" : true
    },
    "time_zone" : {
      "type" : "timezone",
      "length" : 50,
      "not_null" : true
    },
    "email" : {
      "type" : "email",
      "length" : 250,
      "not_null" : true
    },
    "password" : {
      "type" : "password",
      "length" : 60,
      "not_null" : false
    }
  },
  "indexes" : {
    "user_email_index" : {
      "mode" : "unique",
      "fields" : [ "email" ]
    }
  },
  "actions" : { }
}
  
{
  "name" : "group",
  "cdate" : "2015-06-24T17:40:03Z",
  "adate" : "2015-06-24T17:40:03Z",
  "fields" : { },
  "indexes" : { },
  "actions" : { }
}

{
  "name" : "group_user",
  "cdate" : "2015-06-24T17:40:03Z",
  "adate" : "2015-06-24T17:40:03Z",
  "fields" : {
    "group" : {
      "type" : "group",
      "length" : 100,
      "not_null" : true
    },
    "user" : {
      "type" : "user",
      "length" : 100,
      "not_null" : true
    }
  },
  "indexes" : {
    "group_user_index" : {
      "mode" : "unique",
      "fields" : [ "group", "user" ]
    }
  },
  "actions" : { }
}

Using the "allowed_users" and "allowed_groups" configuration parameters in the "permissions.properties" file are indicated the users and groups that have permission to execute actions of the nodes, views and controllers for a certain type.

In order to assign permissions to objects, limit which objects can be referenced or create custom groups you can subclass the Permissions class and configure it with the "permissions" parameter.

user*.permissions=com.nexttypes.settings.UserPermissions

In the system source code you can see the classes UserPermissions or ProjectPermissions as examples. UserPermissions only allows to perform actions on a user and create "user_certificate" objects that refer to that user to administrators and to the user himself. ProjectPermissions limits the actions on "project" and project* objects to the owner of the project and to project members. To make these permission limitations they reimplement the "isAllowed(type,objects,action)" and isAllowedToMakeReference" methods.

Modules Parameters

Each module in the system can have configuration parameters that are saved in a file with the module name and the extension ".properties". The Settings class is used to process these files.

Security

Through the Checks class and the NodeSecurity and ViewSecurity aspects the user permissions are checked to execute the methods of the nodes and views. It is also checked that the names of the types, fields, indexes, identifiers, languages and views comply with the system restrictions to ensure proper operation and avoid code injection. Below is the list of restrictions on names:

• Types, fields, indexes and actions: ASCII characters from "a" to "z", numbers and "_" character.
• Identifiers: ASCII characters from "a" to "z", numbers and "-" and "." characters.
• Languages: ASCII characters from "a" to "z" and "-" character.
• Views: ASCII characters from "a" to "z", numbers and "-" character.

Antivirus

With the ClamAV class, which uses the antivirus of the same name, virus scanning can be performed on the binary, file, document, image, audio and video fields. To activate virus scanning on objects of a type in all or some of its actions the "antivirus" configuration parameter is used.

*.actions.antivirus=false
example.actions.antivirus=true

In the "clamav.properties" config file the "host" and "port" parameters are configured to connect to the ClamAV server.

Activity Registration.

NextTypes using the Logger and FileHandler classes allows you to store logs of system activity from informational messages to critical errors. In the config file "logger.properties " the following parameters can be configured:

nexttypes-2020-12-23.log

If the activity log system fails and the log file cannot be written, as a last resort an error is displayed by the standard output.

Log File Format

Each of the records in the file is made up of two parts. The first part is common to all system modules and includes the date and time in UTC with ISO 8601 format, level of importance, class in which it originated, user who executed the action and IP address from which the action was executed. The second part depends on the module, for example in the HTTP module the method and address of the request are indicated. The PostgreSQL module indicates the SQL query made.

2020-12-22T19:02:23.453Z INFO com.nexttypes.protocol.http.HTTPServlet guest 127.0.0.1 GET http://localhost:8080/
2020-12-22T19:17:13.756Z INFO com.nexttypes.nodes.PostgreSQLNode guest 192.168.1.35 select count(*) as objects, max(udate) as udate from "article"

Node

Node is a storage system inspired by SQL and with a slightly higher level of abstraction than JDBC which provides protection against SQL injection, transaction management and facilitates the basic tasks of creating, reading, modifying and deleting types and objects. It is possible to chain several Node subclasses that process the information until reaching the final node. Each system module has in its configuration the "next_node" parameter that indicates the node to which it must connect.

next_node=com.nexttypes.nodes.ControllersNode

Connection Modes

Nodes can be run in three modes with different privileges to limit the possibility that an attacker can perform illegal actions:

• READ: read only permission.
• WRITE: read and write permission.
• ADMIN: permission to create and drop types.

Execution of queries and actions with protection against SQL injection

The system has the "query" and "execute" methods to perform queries and write actions easily with the execution of a single method. They use prepared statements to separate the values of the columns (fields) from the SQL statement to avoid SQL injection attacks. They also add the possibility of using the names of the tables (types) and the columns (fields) as parameters of prepared statements, using the "#" character, and check that their names are valid and do not present danger of SQL injection. It also offers the possibility of using arrays as parameters.

Tuple[] tuples = nextNode.query("select title from # where id in (?)", type, objects);

Transactions

The Node class implements the AutoCloseable interface, so it can be used in a try-with-resources block.

When a Node object is created a connection to the database is opened with "autocommit" disabled. When the object is closed with the "close" method, it's automatically executed the "commit" method that performs the "commit" in the database. In this way the block try-with-resources corresponds to a transaction.

If you want to save the information in the database at an intermediate point, before closing the connection, you can execute the "commit" method manually. Subtransactions can also be made using "save points".

The "execute" method, before executing a sentence, creates a "savepoint" and makes "rollback" if a failure occurs or the rows affected are not the expected ones.

Concurrency Control

NextTypes uses the types alteration date and objects update date to implement an optimistic concurrency control system. When reading a type or object, the date of the last modification can also be read. When the type or object is updated, that date can be passed to the system in the "alter" and "update" methods. The system checks that the date matches the date of the type or object on the storage system. If the dates do not coincide it is indicative that the type or object has been modified after the last user reading and if your changes are saved the changes made in the other modification will be lost. When this happens the modification is canceled and the user is informed to review the changes that have been made.

Controller

If the default functionality of Node is not enough for some type you can create a controller that adds the desired functionality by creating a subclass of the class Controller and reimplementing the necessary methods. Controllers are run through the ControllersNode node. NextTypes includes various controllers for its basic operation as UserCertificateController and some examples like EmailController.

The controller receives as initialization parameters the type with which it is executed, the authentication information and the next node (nextNode) on which the action is to be performed.

The following example shows the reimplementation of EmailController "insert" method used by the "user_email" type so that when an object of this type is created, in addition to performing the default action in "nextNode", an email is sent using the Email class.

public class EmailController extends Controller {

	public EmailController(String type, Auth auth, Node nextNode) {
		super(type, auth, nextNode);
	}

	@Override
	public ZonedDateTime insert(NXObject object) {
		
		ZonedDateTime udate = nextNode.insert(object);
		new Email(object, Format.HTML).send();
		return udate;
	}
}

Once the controller is created it is assigned to the type with the following configuration line:

user_email.controller=com.nexttypes.controllers.EmailController

Additional Actions

In addition to reimplementing the methods of the Controller class new actions can be added that are executed through the Node "executeAction" methods. A new method is created and marked with the "Action" annotation. Here is the example of ImageController "resize" action.

@Action(RESIZE)
public ActionResult resize(String[] objects, Integer width, Integer height) {
		
}

The actions and the types of their parameters are defined in a JSON file similar to the fields of a type. For example the ImageController uses the image-actions.json file.

{
	"resize": {
		"width": {
			"type": "int32",
			"precision": 32,
			"scale": 0,
			"not_null": true
		},
		"height": {
			"type": "int32",
			"precision": 32,
			"scale": 0,
			"not_null": true
		}
	}
}

In the controller constructor the path of the file is indicated by assigning it to the variable "actionsInfo".

public ImageController(String type, Auth auth, Node nextNode) {
	super(type, auth, nextNode);
		
	actionsInfo = "/com/nexttypes/controllers/image-actions.json";
}

Data format conversion

The controllers also allow you to take data in any format and convert it to an object. The following is an example of the "update" method of the controller ProjectController that uses an event from an iCalendar file to update a "project_meeting" object. This allows a CalDAV client to communicate with the node.

@Override
public ZonedDateTime update(String id, byte[] data) {
	
	if (PROJECT_MEETING.equals(type)) {
		ICalendar calendar = new ICalendar(data);
		VEvent vevent = calendar.getFirstEvent();

  		String startDateString = vevent.getStartDate().getValue();
		String endDateString = vevent.getEndDate().getValue();

		DateTimeFormatter formatter = DateTimeFormatter.ofPattern(Constants.BASIC_DATETIME_FORMAT);

		NXObject object = new NXObject(type, id);
		if (startDateString != null) {
			LocalDateTime startDate = LocalDateTime.parse(startDateString, formatter);
			object.put(KeyWords.DATE, startDate.toLocalDate());
			object.put(KeyWords.START_TIME, startDate.toLocalTime());
		}

		if (endDateString != null) {
			LocalDateTime endDate = LocalDateTime.parse(endDateString, formatter);
				object.put(KeyWords.END_TIME, endDate.toLocalTime());
		}

		object.put(KeyWords.DESCRIPTION, vevent.getDescription().getValue());

		return nextNode.update(object);
	} else {
		return null;
	}
}

Similarities to SQL Triggers

The controllers work similar to the SQL database triggers with some differences:

• They are independent of the database.
• They group all the actions (insert, update, delete, etc).
• They can be assigned and removed with a configuration parameter without the need to execute SQL statements.
• They can be assigned to multiple tables (types).
• They can be applied to read actions (select).

View

Views are subclasses of the View class that connect to Node subclasses in read only mode to display one or more specific types using a data format such as HTML or JSON. For example, an article_language object can be displayed in JSON format, the software_release objects list in RSS format or the project_meeting objects list as an iCalendar file or as an HTML calendar.

Views can also display the forms required to edit the types and objects. Some views instead of directly accessing types and objects take data from SQL queries defined in configuration parameters for each type, converting a SQL query into a view.

The View class implements the AutoCloseable interface so that it can be used in a try-with-resources block. When it is closed, it closes the instance of Node that it has opened. View is an abstract class and only implements the "getVersion" method to display the NextTypes system version and the "notFound" and "unauthorized" methods to show errors. Each subclass of View can implement the following methods:

The views to be used by the different types are configured as follows:

#default views for all types
*.views.html=com.nexttypes.views.HTMLView
*.views.json=com.nexttypes.views.SerialView
*.views.xml=com.nexttypes.views.SerialView
*.views.smile=com.nexttypes.views.SerialView
*.views.webdav=com.nexttypes.views.WebDAVView
*.views.icalendar=com.nexttypes.views.ICalendarView
*.views.rss=com.nexttypes.views.RSSView

#views for type article
article.views.html=com.nexttypes.views.ArticleView
article.views.rss=com.nexttypes.views.ArticleView

#views for type software
software.views.html=com.nexttypes.views.SoftwareView

Content

Once the view is created it is encapsulated in a Content object that is passed to the protocol and contains the following parameters:

application/xhtml+xml;charset=utf-8

example.fields.text.content_type=application/x-latex

"attachment; filename=article.json"

*.content_security_policy=default-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'

Standard Views

NextTypes includes HTML, Serial, RSS, iCalendar, WebDAV, CalDAV, Article and Software views as standard.

HTML

The HTMLView uses HTML5 to show the types and objects as well as the basic forms to create and edit them. In this way it is very easy to add a new type of data to the system. Later, if necessary, the default system can be adapted to the needs that arise.

The HTMLView class implements the following View methods:

Templates

HTMLView uses the HTML class to load an HTML document as a template and modify its DOM in the same way as with Javascript. The templates are stored in the configuration subdirectory "templates". This system allows you to completely separate the HTML design from the programming language code. For the developer the HTML document is another component to use in software development. The template to use for a type is indicated by the "template" parameter.

*.template=default.html

The HTML template should have the following sections:

Below is an example template with the required sections:

<html>

<head>
	<title></title>
</head>

<body>

<header>
	<div id="logo"/>
	<div id="user"/>
	<nav id="langs"/>
	<form id="search"/>
</header>

<nav id="menu"/>
	
<main>
	<noscript/>
	<div id="sidebar">
		<nav id="type-menu"/>
		<nav id="actions"/>
	</div>
</main>

<footer>
	<div id="qrcode"/>
	<div id="validators"/>
</footer>

</body>

</html>

Menus

The control panel links are included in the general menu of the "menu" element if it is activated with the "show_control_panel" variable and the user has permissions to see the destination of those links.

*.show_control_panel=true

Also you can add a custom menu with the information from a JSON file indicating the sections and links for each section. This file is processed by the Menu and MenuSection classes. These files are saved in the "menus" directory of the configuration and configured with the "menu" parameter.

*.menu=menu.json

Each of the sections has a "title" and a series of "archors". Each anchor has a "text" and an "href" address. The titles and texts are variables of the language configuration files that have a text string for each language of the system. For example the variable "project" is assigned the value "Proyecto" in Spanish and "Project" in English.

[{
   "title" : "project",
   "anchors" : [ {
      "text" : "home",
      "href" : "/article/home"
   }, {
      "text" : "system",
      "href" : "/article/system"
   }, {
      "text" : "software",
      "href" : "/software/nexttypes"
   }, {
      "text" : "installation",
      "href" : "/article/installation"
   }, {
      "text" : "contact",
      "href" : "/user"
   } ]
}]

Images

In "image_link" and "image_link_language" objects a list of images is saved that can have a link, a description, an alternative text and a title. "Image_link_language" objects can contain a different image and information for each language and take precedence over "image_link" objects.

To include one of these images in the HTML code it is necessary to include a "div" element with the "image" class and indicate using the custom data attributes "data-id" and "data-lang" the "id" and language of the image.

<div class="image" data-id="system" data-lang="en">

The "images" method of the HTMLView class finds all "div" elements of the HTML document with the class "image" and adds them the corresponding image with an "img" element with alternate text (alt) and an optional title (title). In the "src" attribute the "image" field of the "image_link" or "image_link_language" objects is used.

If the "image_link" or "image_link_language" object has a link the image is inserted in an "a" element with that link. If the object "image_link_language" includes a description the image is inserted in an "figure" element and the description in a "figcaption" element.

<div class="image" data-id="system" data-lang="es">
   <figure>
      <a href="/article/system?lang=es&view=html">
         <img alt="Diagrama Sistema NextTypes"
            src="/image_link_language/e3f7b708-caab-11e4-b8b4-001f3c35b55c/image"
            title="Diagrama Sistema NextTypes"/> 
      </a>
      <figcaption>Diagrama Sistema NextTypes.</figcaption>
   </figure>
</div>

Text Editors

The text fields "text", "html", "json" and "xml" use a "textarea" element and they can have a text editor adapted to their format by using CodeMirror and TinyMCE. By default the "text" fields only have the normal browser editor and the rest of the fields use CodeMirror with one of its modes. The "html" and "xml" fields use the "xml" mode and the "json" fields the "javascript" mode configured to edit JSON.

The Codemirror editor mode of a field can be configured with the "editor" parameter. Furthermore NextTypes adds the "visual" mode to use TinyMCE instead of Codemirror and the "json" mode, which is equivalent to {name: "javascript", json: true}.

user_email.actions.fields.message.editor=visual

The HTMLView class using the "textEditors" method adds to the header of the HTML document the Javascript and CSS files required for TinyMCE, CodeMirror and Codemirror "xml" and "javascript" modes. For the rest of the modes it is necessary to add the necessary files in the template or with a subclass of HTMLView.

<head>
   <script src="/static/lib/codemirror/lib/codemirror.js"/>
   <script src="/static/lib/tinymce/js/tinymce/tinymce.min.js"/>
   <link href="/static/lib/codemirror/lib/codemirror.css" rel="stylesheet" type="text/css"/>
   <script src="/static/lib/codemirror/mode/xml/xml.js"/>
   <script src="/static/lib/codemirror/mode/javascript/javascript.js"/>
</head>

QR Code

The "qrcode" method of the HTMLView class uses the QRCode class to insert in the "div" element with "qrcode" id an image with the QR code of the universal identifier of the displayed object. A data: URL is used, in the PNG format encoded in Base64, so as not to have to expose the QR code generation system to external requests. The "alt" and "title" attributes also include the code value. Here is an example:

<div id="qrcode">
   <img alt="demo.nexttypes.com/article/system"
      src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAFAAAABQCAIAAAABc2X6AAADl0lEQVR4Xu2PQXIEMQgD8/9Pbw5U1rLbwpqt5DKZPlGiAfvr9c/4WoO783z47jwfvjvPh+/O8+G783wYfMWskz+za3rKFZc75n0bAiNmnTx9bE13t1zumPdtCIxgkTrj+A+zu6JOUjsS5/V8eAMPK86p2jHmd1N5l3nVDYGBpYpzqnaM+d1U3mVedUNgYKlCp0+Ys6to15l0GgIDSxU6fcKcXUW7zqTTEBhYqnzmOOa54bNW6DQEBpYqnzmOeW74rBU6DYERLHKHXc2kakLTkTiv58MbxvET6v9FnVB+Q2DEqP8XdUL5DWfjKnqYT2GtSZ//Cn+wUR7Kp7PWpM9/hfNGnudT6PRdmr2jXZdXcuTscSkP0Om7NHtHuy6v5EjqFTyWdKtWXF5wA33NL3FtUo/xsOtWrbi84Ab6ml/iPKkHeGw84QftOkdRx03RcfWRs9evrkTRrnMUddwUHVcfST3CM3yEw025fEx603UXzoaDBzQZT9jhplw+Jr3puguBAVbDO0w0Z8Jcu1o75rkNgQFWwztMNGfCXLtaO+a5DWdD0aU84LqsFeZMNNcuTXW2nA2FZ5Iua4U5E821S1OdLWej4Do9o12tmQz7U1/RrnMWzkbBdfOh0dWaybA/9RXtOmfhbCjh0tfu0XN/4LqVK6uROQtnQwmXvm7wYV3HWhkz3lSHcMp1XdIQe+YRVStjxpvqEE65rksaYg9oTqfvknJ61Lw6+ya19YCeYc2EXVJOj5pXZ9+cbZ6Z++dc0Zym1g6dUvrum8CQRW5pnyua09TaoVNK331zNnL4rKRmUjUdRZ3eXEi9BJ5PaiZV01HU6c2Fs9ev48ned3Bq7N3BKa0bAqNdVF11et/BqbF3B6e0bjgbVxmP8k9xieZ0XM6kIfVy6rw+gg9yieZ0XM6k4ezVugT686Y96utUXzOp+sjZ0wM99OdNe9TXqb5mUvWRs5esc06SV90nCvPeXwiMYJFzkrzqPlGY9/5CYMgiXc18zOweQUdhd8zvUPMS50k9MB9d8zHzrz4891fK6c28O3btki2BgTNK4ijl9GbeHbt2yZbAwBmFjtI7TJi7mgm7WwIDSxU6Su8wYe5qJuxuCYxgEQ9zSnPtrukO7tGkzxcCI1ikTtWc0ly7a7qDezTp84XAiFGfs0miubIau9l5wnL2dGmP+pxNEs2V1djNzhOW1LsNz4fvzvPhu/N8+O48H747z4fvzjdVMRvq+70A4wAAAABJRU5ErkJggg=="
      title="demo.nexttypes.com/article/system"/>
</div>

Calendar

The HTML view includes the ability to view the list of objects of a type such as a calendar. The view takes as parameters the year and the month. The calendar is created from the parameters "id", "summary", "date", "start_time" and "color" of the query indicated in the configuration parameter "calendar.select".

project_meeting.calendar.select=select pm.id, concat(p.name,': ',pm.summary) as summary, pm.date, pm.start_time, p.color from project p join project_meeting pm on p.id=pm.project

Serial

The SerialView implements various View methods and uses the Serial class to display the information in JSON, XML and Smile formats. This view is used to remotely access the data, for example with Javascript.

RSS

The RSSView implements the "select" method of View and uses the RSS class to display the type objects list in RSS format.

The elements of the RSS feed are formed from the "id", "title", "description" and "pub_date" parameters of the results of an SQL query indicated in the configuration parameter "views.rss.select". The following is an example of the query used for the "software" type:

software.views.rss.select=select s.id, concat(sl.title,' ',sr.version) as title, left(sl.description,400) as description, sr.cdate as pub_date from software s left join software_language sl on (s.id=sl.software and sl.language=?) left join software_release sr on s.id=sr.software order by sr.cdate desc

When a type has the "views.rss.select" parameter configured, in the HTML view a "link" element is added in the header with the address of the RSS feed of the object or of the list of objects that is being displayed. In the type menu a link with the same address is added.

<link href="/article?lang=es&view=rss" rel="alternate" title="RSS" type="application/rss+xml"/>

If you need to create the view from other parameters you can create a subclass of View and assign it as an RSS view.

article.views.rss=com.nexttypes.views.ArticleView

ICalendar

The ICalendarView implements the "select" method of View using the ICalendar class to display the type objects list in iCalendar format.

The calendar is formed by taking the "id", "summary", "start_date", "end_date" and "description" parameters of the results of the SQL query indicated in the configuration parameter "views.icalendar.select".

project_meeting.views.icalendar.select=select pm.id, pm.udate, concat(p.name,': ',pm.summary) as summary, pm.description, (pm.date+pm.start_time) as start_date, (pm.date+pm.end_time) as end_date from project p join project_meeting pm on p.id=pm.project

When a type has the "views.icalendar.select" parameter configured, in the HTML view a link to the iCalendar view is added in the type menu. If you need to create the view from other parameters you can create a subclass of View and assign it as an iCalendar view.

With this view you can see the objects on a client that can read iCalendar files, such as Thunderbird.

WebDAV

The WebDAVView implements the "getTypesName", "select", "get" and "getField" methods to display the list of types, list of objects of a type, view of an object and the fields of the object in WebDAV format to be used from the HTTP/WebDAV protocol.

This view is used so that the hierarchy of types, objects, and fields can be viewed as a file system. Types and objects are viewed as folders and fields as files that can be viewed and edited with any program that can connect through WebDAV. For example you can explore the system with a file manager like Caja, edit a text field with a text editor such as Pluma or modify a document with LibreOffice.

CalDAV

The CalDAVView reimplements the WebDAV "select" method to display the list of objects of a type in CalDAV format with "getcontenttype" "text/calendar" when used the PROPFIND HTTP method and as calendar-data resources containing an iCalendar calendar when used the REPORT HTTP method. Examples are shown below:

<?xml version="1.0" encoding="UTF-8"?>
<D:multistatus xmlns:D="DAV:">
   <D:response>
      <D:href>/project_meeting/</D:href>
         <D:propstat>
            <D:prop>
               <D:getetag/>
               <D:getcontenttype/>
            </D:prop>
               <D:status>HTTP/1.1 404 Not Found</D:status>
            </D:propstat>
            <D:propstat>
               <D:prop>
                  <D:resourcetype>
                     <calendar xmlns="urn:ietf:params:xml:ns:caldav"/>
                     <D:collection/>
                  </D:resourcetype>
               </D:prop>
               <D:status>HTTP/1.1 200 OK</D:status>
            </D:propstat>
            <D:responsedescription/>
         </D:response>
         <D:response>
            <D:href>/project_meeting/25c4b5fe-1908-11e4-8434-001eec2e2304</D:href>
            <D:propstat>
               <D:prop>
                  <D:getetag>887ed13bc8d6837f0d22108cfc27d60e</D:getetag>
                  <D:getcontenttype>text/calendar</D:getcontenttype>
                  <D:resourcetype/>
               </D:prop>
               <D:status>HTTP/1.1 200 OK</D:status>
            </D:propstat>
            <D:responsedescription/>
         </D:response>
...

<?xml version="1.0" encoding="UTF-8"?>
   <D:multistatus xmlns:D="DAV:">
      <D:response>
         <D:href>/project_meeting/44de15f9-e801-4b08-a7ce-74d724bc0629</D:href>
         <D:propstat>
            <D:prop>
               <D:getetag>be02265e4f0a1c84cf74a541c7dfd18a</D:getetag>
               <calendar-data xmlns="urn:ietf:params:xml:ns:caldav">
BEGIN:VCALENDAR
PRODID:nexttypes
VERSION:2.0
CALSCALE:GREGORIAN
BEGIN:VEVENT
DTSTAMP:20201219T205704Z
DTSTART:20201229T170000
DTEND:20201229T200000
SUMMARY:Project 1: Meeting 8
DESCRIPTION:Meeting 8
ATTACH:https://demo.nexttypes.com:8443/project_meeting/44de15f9-e801-4b08-a7ce-74d724bc0629
UID:dd503687-4796-4517-9e42-17e19ebf02c7
END:VEVENT
END:VCALENDAR
               </calendar-data>
            </D:prop>
            <D:status>HTTP/1.1 200 OK</D:status>
         </D:propstat>
         <D:responsedescription/>
      </D:response>
...

Calendars are created with the same information as in the ICalendar view and can be viewed and edited by programs that use CalDAV.

Protocol

The protocols are modules that allow the execution of the Node, Controller and View methods through data transmission protocols. The protocols currently implemented and included in NextTypes are HTTP and SMTP.

HTTP

This implementation integrates NextTypes with the most used protocols on the Internet, allowing to integrate NextTypes with a multitude of current systems since these protocols are implemented in all web browsers, support for editing files through WebDAV is present in all operating systems and there are HTTP libraries for all programming languages.

The implementation consists of using the GET, POST and PUT HTTP methods to execute the methods defined in View and Node following a REST architecture in which all types, objects, fields of the objects and elements of the fields can be accessed through a URL and they can have different representations.

The system consists of the HTTPServlet and a series of auxiliary classes to process the requests. Its configuration is in the "http.properties" file. HTTPServlet receives all requests except for those directed to "/static", which directly access static files.

Each HTTP request instantiates a subclass of Node or View with a try-with-resources block. In this way each HTTP request corresponds to a Node transaction. In the case of a write request, if an error occurs, Node "rollback" method is automatically executed undoing the changes made.

URL

The format of the URL is as follows:

protocol://server:port/type/id/field/element?parameter1=&parameter2=

The URL is made up of the following levels:

Different parameters can be applied to each URL that make the behavior of the system vary. NextTypes specifies some of these parameters and others can be added to the views as needed. Below is a table with the standard parameters and their description:

default_lang = en

default_view = html

/tipo?form=insert

View Methods

View methods are read operations, which is called in HTTP Safe Methods, and made through an HTTP GET request to a URL with the format indicated above. Each of the levels of the URL in combination with the appropriate parameters executes a method of the View class which in turn executes one or more methods of the class Node in read only mode to read the data needed to build the view and pass it to the protocol in a Content object.

The following is the list of URL addresses and their correspondence to View methods:

index = /?info

Node Methods

Requests must be directed to the URL of the type or object on which the action must be applied. The POST and PUT methods are used to perform write operations, except for the export of types and objects.

In the HTTP session is stored a random string of characters that must be sent in POST requests in the "session" parameter to prevent CSRF attacks.

Requests with the POST method are in the multipart/form-data or application/x-www-form-urlencoded formats. This allows you to send the parameters from an HTML form or using any HTTP library without the need to build NextTypes objects. It is the server that is responsible for building the object.

The PUT method is used to update objects and fields by WebDAV clients. Clients can directly edit a field. They can also edit an object or field through a controller. For example a text editor can directly edit a "document" field or a calendar program update an object with an iCalendar file using a controller.

The "_action" parameter is used to indicate the method of Node or the additional action of a controller to be executed, since several actions are executed on the same URL. The "action" parameter is not used because the name conflicts with the "action" attribute of the HTML form.

In the insertion and updating of objects as well as in the execution of additional actions the names of the fields are preceded by the "@" character so that their name does not coincide with system parameters such as "_action" or "session".

Below is a list of actions, URLs and required parameters:

Coding of Lists of Composite Elements.

In the creation and alteration of types it is necessary to send lists of fields and indexes that are composed of several parameters. To do this the name of each parameter consists of list name, position of the field or index within the list and the parameter name separated by ":".

list:position:parameter

For example the following shows the parameters of a type with two fields and an index.

fields:0:type
fields:0:name
fields:0:parameters
fields:0:not_null
  
fields:1:type
fields:1:name
fields:1:parameters
fields:1:not_null
  
indexes:0:mode
indexes:0:name
indexes:0:fields

Filters for objects lists are also made up of several parameters.

filters:0:field
filters:0:comparison
filters:0:value

Security

Connections are made using HTTPS. The server must have a digital certificate. When a client connects using HTTP it is redirected to use HTTPS on the port indicated in the "https_port" variable. The session cookie is set in the file WEB-INF/web.xml as "Secure" so that it is only sent over HTTPS and as "HTTP Only" so that it is not accessible from Javascript.

<session-config>
   <session-timeout>30</session-timeout>
   <cookie-config>
      <http-only>true</http-only>
      <secure>true</secure>
   </cookie-config>
</session-config>

The HTTPRequest class reads the information from the HTTP request and checks that all parameters meet system restrictions.

Authentication

Through the secure channel of the HTTPS connection the authentication of a user can be done in two ways: passwords and digital certificates. In turn password authentication can be done through the POST method or with Basic HTTP Authentication. The password is checked using the Node.checkPassword method.

When a user authenticates the system reads the groups to which he belongs with the Node.getGroups method and saves an Auth class with the user and groups in the HTTP session.

To counteract brute force attacks the system limits the number of authentication failures from the same IP address for one minute with the "max_auth_errors" configuration parameter.

max_auth_errors=5

POST Method

The HTTPServlet has two methods of logging in and logging out. As with Node the method to use is indicated by the "_action" parameter.

Digital Certificates

Clients can connect presenting a X.509 digital certificate to the server. Through objects of user_certificate type the server saves a list of the "subjects" of the certificates that allow the authentication of a user. When a user connects with a certificate the system looks for the user to whom the certificate belongs.

{
  "name" : "user_certificate",
  "cdate" : "2015-06-24T17:40:03Z",
  "adate" : "2015-06-24T17:40:03Z",
  "fields" : {
    "user" : {
      "type" : "user",
      "length" : 100,
      "not_null" : true
    },
    "certificate_subject" : {
      "type" : "string",
      "length" : 250,
      "not_null" : true
    }
  },
  "indexes" : {
    "user_certificate_subject_index" : {
      "mode" : "unique",
      "fields" : [ "certificate_subject" ]
    }
  }
}

Basic HTTP Authentication

This system is included for use by programs that cannot use the other two systems. Using the "401 Unauthorized" status code and the "WWW-Authenticate" header the client is asked to use this authentication system. The "basic_auth_realm" configuration parameter sets the name of the protection space for which authentication is required. By default NextTypes only has one space that covers the entire system.

basic_auth_realm=NextTypes

This protection space is indicated to the client in the "WWW-Authenticate" header.

WWW-Authenticate: Basic realm="NextTypes"

The "basic_auth_user_agents" variable configures the clients that are going to be asked for this type of authentication.

basic_auth_user_agents=LibreOffice,gvfs,cadaver,kioexec,Thunderbird

When one of these clients wants to carry out an action for which he needs to be authenticated and have permission he must include in the request the "Authorization" header with the username and password separated by ":" and encoded in Base64.

authorization: Basic YWRtaW46QWRtaW4jNDQ=

In Linux with the "base64" command we can decode it to see the username and password.

echo "YWRtaW46QWRtaW4jNDQ=" | base64 -d

admin:Admin#44

Limitation of Requests per Minute

To protect against denial of service attacks the number of requests per minute from the same IP address are limited. Total requests are limited by the "max_requests" parameter of "http.properties" and the maximum number of inserts of objects of a type with the "max_inserts" parameter of "types.properties".

max_requests=150
*.max_inserts=5

WebDAV

When a request is received with the PROPFIND or REPORT HTTP methods, it is processed by the view indicated in the "views.webdav" configuration parameter.

*.views.webdav=com.nexttypes.views.WebDAVView
project_meeting.views.webdav=com.nexttypes.views.CalDAVView

NextTypes includes the WebDAVView, which displays types, objects, and items such as directories and files; and the CalDAVView that displays a list of objects as calendar events. In addition other views can be created that show the objects in other formats used through WebDAV.

Reading and writing data in WebDAV is done with the GET and PUT methods that are processed by the View and Node methods.

Status Codes

For the response to the client informing the result of an action the HTTP status codes are used, together with result data or a text string describing the result:

host=demo.nexttypes.com

index=/?info

SMTP

The SMTPServlet partially implements a SMTP mail server with the basic functions to receive emails from a complete mail server. The server receives the email messages and converts them to raw_email objects.

{
  "name" : "raw_email",
  "cdate" : "2015-04-01T14:30:16Z",
  "adate" : "2015-04-01T14:30:16Z",
  "fields" : {
    "mail_from" : {
      "type" : "string",
      "length" : 250,
      "not_null" : true
    },
    "rcpt_to" : {
      "type" : "text",
      "not_null" : true
    },
    "data" : {
      "type" : "text",
      "not_null" : true
    }
  }
}

By creating a controller for this type it is possible to process these objects and perform actions with their data or convert them to another type.

The server is configured with the "smp.properties" file and the following configuration parameters:

220 demo.nexttypes.com NextTypes SMTP

Tasks

The TasksServlet allows executing background tasks that perform different actions such as backups, scanning of virus or sending emails.

Each of the tasks is a subclass of the Task class and are added to the "tasks" parameter of the "tasks.properties" configuration file separated by commas.

tasks=com.nexttypes.system.BackupTask

Backups

NextTypes includes the BackupTask that performs incremental and full backups. In the "backup.properties" configuration file the following parameters are defined:

The backup files are types containers with the ".json" extension and have in their name the date and time when the backup was made. At the beginning of the name the prefix indicated in the variable "prefix" is added and at the end "incr" or "full" if the copy is incremental or full.

nexttypes-2021-03-29T03:15:15.294936Z-incr.json
nexttypes-2021-03-29T11:15:15.416129Z-full.json

When this task starts it analyzes the files in the directory indicated by the "directory" variable to find out how many incremental copies have been made and the date of the last one. With this information when the time indicated in the "interval" variable has passed since the last copy a new copy is made. If the number of incremental copies indicated in "incremental" has been reached a full copy is created.

To perform the incremental copy executes the node "backup" method with "full" parameter with "false" value. The node returns all the objects that have the "backup" property with "false" value, which indicates that they have not been backed up since their last update. To make a complete copy the node "backup" method is executed with "full" parameter with "true" value. In this case the node returns all the objects. Once the copy is made the "backup" property of the objects is updated to "true".

Console

NextTypes includes the Console class that allows you to perform some actions on the system from command console without using a protocol such as HTTP or the HTML graphical interface. To run it you need to go to the directory where the WAR file has been unzipped and indicate with the "-cp" parameter the location of the classes and the JAR files of the system.

With the Console "-h" parameter help is shown with the available actions and their parameters. The "-s" parameter indicates the directory where the settings we want to use to access the system are located.

cd /var/lib/tomcat9/nexttypes.com/ROOT/WEB-INF

java -cp classes:lib/* com.nexttypes.system.Console -h
usage: com.nexttypes.system.Console -b | -eo <objects> | -et <types> | -h
       | -io | -it | -vs  [-eoa <action>]  [-eta <action>] [-f]  [-ino]
       [-l <lang>] [-o <objects>] [-or <order>] [-s <settings>] [-t
       <type>]
 -b,--backup                               Backup types and objects.
 -eo,--export-objects <objects>            Export objects.
 -eoa,--existing-objects-action <action>   Existing objects action.
 -et,--export-types <types>                Export types.
 -eta,--existing-types-action <action>     Existing types action.
 -f,--full                                 Make a full backup.
 -h,--help                                 Help.
 -ino,--include-objects                    Include objects.
 -io,--import-objects                      Import objects.
 -it,--import-types                        Import types.
 -l,--lang <lang>                          Language.
 -o,--objects <objects>                    Objects Id.
 -or,--order <order>                       Query order.
 -s,--settings <settings>                  Settings directory.
 -t,--type <type>                          Type name.
 -vs,--virus-scan                          Virus scan.

For example with the "-et" parameter you can export a list of types. If no type is indicated all are exported. Adding the option "-ino" includes the objects of the types. The types are written in the "standard output" and can be redirected to a file or another program that processes the types.

java -cp classes:lib/* com.nexttypes.system.Console -s /etc/nexttypes/settings/ -et -ino > types.json

For the import of types the reverse operation is carried out, the file containing the types is redirected to the program "standard input".

java -cp classes:lib/* com.nexttypes.system.Console -s /etc/nexttypes/settings/ -it -eta abort -eoa abort < types.json

With the "-b" parameter a backup can be made. The "-f" parameter indicates that the copy is full. This allows us to make backup copies with a planning system external to the system such as "cron" or "systemd".

java -cp classes:lib/* com.nexttypes.system.Console -s /etc/nexttypes/settings/ -b > types-backup.json

The "-vs" parameter is used to scan viruses. The parameter "-t" indicates the type and the optional parameter "-o" the objects to be scanned. If the "-o" parameter is omitted all objects of the type are scanned.

java -cp classes:lib/* com.nexttypes.system.Console -s /etc/nexttypes/settings/ -vs -t example

example::test-document: Virus Found: binary -> Win.Test.EICAR_HDB-1