DocDB

DocDB module installation and uninstallation guide

Installation

Prerequisite

If you have prior experience with command-line operations, you should succeed without difficulty (it shouldn't take you more than ten minutes). If you have successfully installed Dolibarr, installing the docDB module should not be a problem for you.
However, in case of any issues, you will need some integrator experience.
Please note that the commands can be tested before being executed.
IN ALL CASES: It is absolutely essential to backup your Dolibarr folder, documents, and database BEFORE starting.

DocDB has been tested in the following contexts, but that does not mean it won't work on other versions or systems. It is developed with common functions. If you have sufficient knowledge of Dolibarr components, feel free to try it, and if everything works perfectly during the testing phase, it is unlikely that errors will occur later on.
If you have developed a specific module or program, refer to the section dedicated to that topic further in this document.

Dolibarr Version 17.0.0
Database Systems Mysql or MariaDB, any version used by Dolibarr
PostgresSQL not supported
Operating Systems Linux, all versions supporting Python 3
Windows, all versions starting from 10
(You will need to install a standard Python 3 version. Please note that the version adapted by Microsoft may cause numerous issues.)
Python Version 3.9 and later
Python mysql.connector Version 8.0.32 and later

Operating mode

If you have prior command-line experience, you should succeed without difficulty (it shouldn't take you more than ten minutes). If you have successfully installed Dolibarr, installing docDB should not be a problem for you.
However, in case of any issues, you will need a minimum level of integrator experience.
Please note that the commands can be tested before actually executing them.
IN ALL CASES: It is absolutely essential to back up your Dolibarr folder, documents, and database BEFORE starting.

The docDB module is preferable as it provides a document framework, but it is not strictly necessary. In other words, you can copy the programs by other means and activate them without installing the module.
The installation process starts modifying Dolibarr scripts, and then copies the documents into the database.

> Python installation

For Linux, Python is included in the system.
Verification:
$ python3 -V
Python 3.9.2
For Windows, use the automatic installer of the latest version obtained from www.python.org/downloads/, which takes a few minutes. Don't forget to include Python in the PATH (indicated as Windows environment in the installer).
Verification:
$ py -V
Python 3.11.3
If you get a different version number, it doesn't matter as long as it's a version 3 (the scripts are not compatible with version 2, which is discontinued at this time).
Microsoft provides Python, but it's a modified version not recommended by your humble servant. Do not use it, as it's not up to date and brings numerous issues.
To run the command after installing the official version, use py instead of python3. This applies to everything that follows.

> Mysql connector installation

On Linux, initiate the installation of the module and confirm.
$ apt install python3-pip
$ pip3 install mysql-connector-python
To verify the installation:
$ python3
>>> import mysql.connector
On Windows, download the installer from dev.mysql.com/downloads/connector/python/ and install it. You will also need to install the Microsoft VC redistributable package.
$ py
>>> import mysql.connector

> Php installation

Normally, PHP is already installed since Dolibarr is functioning. The file migration command requires PHP to find the characteristics of certain documents (Python could do it, but the results it would get might be slightly different).
In the php.ini configuration file (e.g., in /etc/php/7.4/apache2/php.ini), the exif extension (extension=exif) must be enabled, which is usually the case. You should only modify it if it causes crashes.

> Installation of the docDB package

Retrieve the package from the Dolistore and follow the standard package installation procedure.

> Web server stopping

In principle, we are going to modify Dolibarr, and it is not relevant for it to continue running. Therefore, it is imperative to stop the web server. However, the database can and must continue to function.
In a terminal:
# systemctl stop apache2
Using a command prompt launched as an administrator:
> net stop Apache2.4
Before continuing, as a precaution, check that Dolibarr is inaccessible.

> Changing to the right directory

Let's assume that Dolibarr is installed in the directory /var/www/dolibarr/htdocs (we will use this example later on). You need to navigate to the install directory of the module. For Windows, replace the / characters with \. We will use the directory \Apache24\htdocs\dolibarr
Note that folders outside of htdocs are not considered. This is a limitation that should not bother you.
$ cd /var/www/dolibarr/htdocs
$ cd custom/docdb/install
$ cd \Apache24\htdocs\dolibarr
$ cd custom\docdb\install
If the second command does not work, it means that the module is not installed.

> Dolibarr scripts conversion

$ python3 docDBmigrateScripts.py --dry-run --dolibarrDir=/var/www/dolibarr/htdocs
The -X utf8 switch is necessary if the default character set is not Unicode (which is often the case).
$ py -X utf8 docDBmigrateScripts.py --dry-run --dolibarrDir=\Apache24\htdocs\dolibarr
You are requesting the execution of docDBmigrateScripts.py, with the --dry-run switch, specifying the location of Dolibarr as /var/www/dolibarr/htdocs (or \Apache24\htdocs\dolibarr). The program checks that it finds Dolibarr at this location, that the documents folder exists, and that the configuration of the source files is correct. Then it asks for confirmation:
$ python3 docDBmigrateScripts.py --dry-run --dolibarrDir=/var/www/dolibarr/htdocs
-- checking /var/www/dolibarr/htdocs
version Dolibarr 16.0.5 found at the right place
-- checking configuration file /var/www/dolibarr/htdocs/conf/conf.php
configuration file /var/www/dolibarr/htdocs/conf/conf.php is good
-- checking documents path /var/dolibarr/documents
directory /var/dolibarr/documents exists (must be writable, I did not check it)
-- checking source files referenced in ./scriptFiles/16.0.5
source files referenced in ./scriptFiles/16.0.5 are good
>> please confirm (y/N) I must migrate this directory: /var/www/dolibarr/htdocs
NB: this is a dry run, I am just showing what could happen
Confirmation is requested (type y). In any case, since you specified --dry-run, nothing will be modified.
Take care ! If you have modified certain Dolibarr sources (their location is indicated in the --checking source files referenced step), or if their version is different from the expected version, this will not work and the program will not proceed. It will display something like:
some source file(s) referenced in ./scriptFiles/16.0.5 are not OK
nothing was done, see you later
If you have confirmed, the program displays:
-- migrating /var/www/dolibarr/htdocs
../scripts/user/sync_groups_ldap2dolibarr.php ['fgets']
../scripts/user/sync_users_ldap2dolibarr.php ['fgets']
../scripts/user/migrate_picture_path.php ['is_dir', 'opendir', 'readdir']
../scripts/website/migrate-news-joomla2dolibarr.php ['file_get_contents']
../scripts/product/migrate_picture_path.php ['is_dir', 'opendir', 'readdir']
  ...
../htdocs/hrm/class/skillrank.class.php ['file_exists', 'rename']
../htdocs/hrm/class/skill.class.php ['file_exists', 'rename']
../htdocs/hrm/class/position.class.php ['file_exists', 'rename']
-- updating configuration file
-- updating source files referenced in ./scriptFiles/16.0.5
-- copying filebypass.php
migration should not act as expected (578 files, 3589 changes; would migrate but 574/3558 expected)
It displays the list of numerous scripts to modify, as well as the PHP functions to modify in brackets. In the example below, the modification of the 'is_dir', 'opendir', 'readdir' functions is anticipated in the migrate_picture_path.php script.
Note the phrase migration should not act as expected. It means that the number of scripts is not as expected. This happens when a module (or more) has been installed. The program only knows the number of standard scripts, which is why it indicates a difference. It is not important, the work will be done as long as we remove --dry-run in our command. Let's proceed:
$ python3 docDBmigrateScripts.py --dolibarrDir=/var/www/dolibarr/htdocs
$ py -X utf8 docDBmigrateScripts.py --dolibarrDir=\Apache24\htdocs\dolibarr
In our example, we get the following result:
migration not done as expected (578 files, 3589 changes; migrated but 574/3558 expected)
The important word is migrated, which means that the work has been done.
At this stage, if we were to consult Dolibarr, we would think that our documents have disappeared. No worries, they would still be in the same location, but since Dolibarr would be accessing the database instead of the file system, and the database would not contain any documents, it would show errors.

> Documents conversion

The method to convert documents is analogous to that used for PHP scripts. A command to be run (docDBmigrateFiles.py this time), with checks, a confirmation, and a result. Let's try with --dry-run. Note that it is the Dolibarr directory that is given as an argument, not the Dolibarr documents directory. We understand that access to the configuration file is necessary to connect to the database.
$ python3 docDBmigrateFiles.py --dry-run --dolibarrDir=/var/www/dolibarr/htdocs
If PHP is not in the path, the use of the switch --phpdir is necessary.
$ py -X utf8 docDBmigrateFiles.py --dry-run --dolibarrDir=\Apache24\htdocs\dolibarr --phpdir=\php
The result with the request for confirmation:
-- checking /var/www/dolibarr/htdocs
-- checking configuration file /var/www/dolibarr/htdocs/conf/conf.php
configuration file /var/www/dolibarr/htdocs/conf/conf.php is good
-- checking documents path /var/dolibarr/documents
directory /var/dolibarr/documents exists
-- checking connection on database docDB16
connected on database docDB16, reading documents information
15 documents referenced in the database, thumbs included
>> please confirm (y/N) I must migrate this directory : /var/www/dolibarr/htdocs
NB: this is a dry run, I am just showing what could happen
The program indicates in our case 15 documents to migrate. Note that a newly installed Dolibarr contains no documents (in this case the message will be: 0 documents referenced in the database).
Let's confirm to see.
-- migrating documents
/facture/FA2304-0001/FA2304-0001-02 - west run - part 1 - dscf3650.jpg
/facture/FA2304-0001/FA2304-0001-078E-Auguste-Herbin-moulin-o.jpg
 ...
/mycompany/logos/thumbs/apia-asso-logo-carre_mini.png
/mycompany/logos/thumbs/apia-asso-logo-carre_small.png
/mycompany/logos/thumbs/apia-asso-logo_mini.png
/mycompany/logos/thumbs/apia-asso-logo_small.png
documents migration should act as expected
The last line tells us that everything should go well. Of course, --dry-run is absolutely not mandatory, and we could have executed the command directly.
$ python3 docDBmigrateFiles.py --dolibarrDir=/var/www/dolibarr/htdocs
$ py -X utf8 docDBmigrateFiles.py --dolibarrDir=\Apache24\htdocs\dolibarr --phpdir=\php
With the result:
-- checking /var/www/dolibarr/htdocs
-- checking configuration file /var/www/dolibarr/htdocs/conf/conf.php
configuration file /var/www/dolibarr/htdocs/conf/conf.php is good
-- checking documents path /var/dolibarr/documents
directory /var/dolibarr/documents exists
-- checking connection on database docDB16
connected on database docDB16, reading documents information
15 documents referenced in the database, thumbs included
>> please confirm (y/N) I must migrate this directory : /var/www/dolibarr/htdocs
y
-- migrating documents
/facture/FA2304-0001/FA2304-0001-02 - west run - part 1 - dscf3650.jpg
/facture/FA2304-0001/FA2304-0001-078E-Auguste-Herbin-moulin-o.jpg
/facture/FA2304-0001/FA2304-0001.pdf
  ...
/mycompany/logos/thumbs/apia-asso-logo-carre_small.png
/mycompany/logos/thumbs/apia-asso-logo_mini.png
/mycompany/logos/thumbs/apia-asso-logo_small.png
-- creating directories
documents migration done, you can move some /var/dolibarr/documents subdirectories elsewhere to keep them safe, but please don't move the following (because Dolibarr need them to work properly) :
    /api/temp
    dolibarr.log
    install.lock
This time the migration is complete (documents migration done). The program suggests moving the documents folder. This will confirm during the operation of Dolibarr that the access is done correctly in the database and not in the usual location where they are no longer present. In any case, make sure to keep the indicated folders and files in the right place, otherwise the operation of Dolibarr would be greatly disrupted.
Furthermore, there is mention of creating directories. What is this about since the documents are now in a database? It is simply that docDB emulates a standard file behavior and requires a table (llx_doc_directory) that contains the names of the folders supposed to contain the documents. In reality, the documents are stored in another table (llx_doc_data) that serves as a container. You can connect to MySQL and execute queries to see how it works:
MariaDB [docDB16]> select * from llx_doc_directory;
+-------+-----------------------------+
| rowid | path_name                   |
+-------+-----------------------------+
|     1 | /admin                      |
|     2 | /admin/temp                 |
|     3 | /admin/temp/                |
  ...
 MariaDB [docDB16]> select rowid,path_name,filemtime,octet_length(datablob) as filesize,octet_length(exif_data) as exif_data,
octet_length(imagesize) as imagesize from llx_doc_data order by rowid;
+-------+-----------------------------------------------------------------------------------------------------+---------------------+----------+-----------+-----------+
| rowid | path_name                                                                                           | filemtime           | filesize | exif_data | imagesize |
+-------+-----------------------------------------------------------------------------------------------------+---------------------+----------+-----------+-----------+
|     1 | /facture/FA2304-0001/FA2304-0001-02 - west run - part 1 - dscf3650.jpg                              | 2023-04-08 11:01:07 |  1813708 |      4184 |       137 |
|     2 | /facture/FA2304-0001/FA2304-0001-078E-Auguste-Herbin-moulin-o.jpg                                   | 2023-04-08 11:01:17 |   205915 |       783 |       133 |
|     3 | /facture/FA2304-0001/FA2304-0001.pdf                                                                | 2023-04-11 11:16:27 |    45379 |         4 |         4 |
  ...
Of course, the useful parts of the documents are not displayed here.
Also, if Dolibarr has just been installed, you will get either Empty set or Table doesn't exist after each of the two queries.
For further explanation on how these tables work, please visit the Dolibarr wiki.

> Ping

Please tell me you installed docDB. This will give me an idea about the number of people insterested by this module.
$ python3 docDBping.py
$ py -X docDBping.py

> Web server starting

A simple command :
# systemctl start apache2
> net start Apache2.4
That's it, it's done.
Don't forget to adjust your backup procedure. Simply remove the part concerning the directory that contained the documents. As for the database, nothing changes, except that the backup will be larger and take longer.
One last comment: the effort required to switch to Linux is not that significant, and the return on investment is so quick!

Coming back to the standard state (uninstallation)

Let's imagine that after a trial period, the docDB solution doesn't suit your needs. Let's say you've changed your mind and would like to revert to the standard state. By the way, I would love to know why, feel free to let me know :-)
This is possible, simply replay the exact same procedure, but add --reverse to the docDBmigrateScripts.py and docDBmigrateFiles.py commands. This would be useful if Dolibarr users have added data, in which case the backups you made before the migration would not be up to date.
$ python3 docDBmigrateScripts.py --reverse --dolibarrDir=/var/www/dolibarr/htdocs
  ...
$ python3 docDBmigrateFiles.py --reverse --dolibarrDir=/var/www/dolibarr/htdocs
  ...
$ py -X utf8 docDBmigrateScripts.py --reverse --dolibarrDir=\Apache24\htdocs\dolibarr
  ...
$ py -X utf8 docDBmigrateFiles.py --reverse --dolibarrDir=\Apache24\htdocs\dolibarr --phpdir=\php
  ...
It goes without saying that you can also use --dry-run in this operation.
You would have stopped the web server before the operation, and you will probably need to modify the access rights of the documents after the operation (the login you are using will probably not grant access to the Apache user). The necessary commands will be provided at the end of the operation. And don't forget to delete the llx_doc_directory and llx_doc_data tables in the database after the operation.
documents REVERSE migration done
take care about access rights of your documents ! If www-data is your web server user, I suggest :
    chown -R www-data:www-data /var/dolibarr/documents
    chmod u+w -R /var/dolibarr/documents
You should also clean up your database docDB16 :
    drop table llx_doc_directory;
    drop table llx_doc_data;

In case of Dolibarr update

What happens in this case? Well, it's very simple: you need to update your Dolibarr scripts as usual in the folder on the web server, and before making it work (that is, before running the update within Dolibarr), run the docDBmigrateScripts.py command again. Otherwise, Dolibarr won't be able to find its documents. Note that docDBmigrateScripts.py can be replayed multiple times without any inconvenience. If it doesn't find any PHP scripts to migrate, it will simply do nothing. This is what happens if you run it twice in a row.
There is, however, a nuance: some specific scripts (those located in the install/scriptFiles folder) will be systematically copied. These are the ones for which automatic migration was too risky to program. There is no impact if they haven't changed in the new version of Dolibarr. Otherwise, it is better to wait for the version of DocDB that will take into account this new version of Dolibarr.
On the other hand, docDBmigrateFiles.py should be used with caution because it overwrites the content of the documents in the database. If users have uploaded or generated documents, they may be lost. Using this command is completely unnecessary and almost certainly harmful once you have started using Dolibarr with docDB.

If you add a Dolibarr module

If your module is installed in Dolibarr before installing DocDB, nothing special needs to be done. The conversion program will take care of the additional scripts and convert the functions related to the documents.
If you install your module afterwards, proceed as usual but run the conversion program again, initially in dry run mode. It will indicate the scripts that have not been converted and will convert them if you confirm.
If your module does not specifically handle documents, it will not be affected. Otherwise, test it thoroughly before putting it into production.