When you share a dataset with D2xlab, the application automatically compresses it before uploading it to the server. The goal is simple: reduce storage requirements and speed up data transfers while preserving virtually all of the useful information.

In this article, we’ll explain how this compression works, why a traditional lossless approach is not sufficient for numerical time series, and how D2xlab achieves significantly better compression with only a negligible impact on most datasets.

To illustrate the process, we’ll use a recording of a smartphone’s 3-axis accelerometer and gyroscope placed on top of a washing machine—a surprisingly rich dataset full of interesting features.

How D2xlab processes your data

Internally, D2xlab performs all numerical computations using 64-bit floating-point values (double precision).

This is the standard choice for scientific computing because it minimises rounding errors during numerical operations such as filtering, integration, differentiation, interpolation or spectral analysis. Whether running in a browser or on a desktop application, using double precision ensures that numerical accuracy is preserved throughout the processing pipeline.

When you import a dataset into D2xlab, regardless of its original format, every value is converted to a 64-bit floating-point number. This does not mean your original data suddenly becomes more accurate. For example, if a sensor effectively provides only 12 bits of meaningful information, converting it to 64-bit precision simply pads the value with many meaningless bits. That’s perfectly fine: the internal representation is optimised for computation, not for storage.

Efficient storage

One of D2xlab’s strengths is how easily datasets can be shared.

Before generating a shareable link, the application uploads your dataset to the server, where it is stored until needed. Larger datasets require more storage space and take longer to transfer over the network, so reducing their size directly improves both storage costs and the user experience. Compression is therefore an important part of the sharing process.

Broadly speaking, there are two categories of compression algorithms:

• Lossless compression, where the original data can be recovered exactly.
• Lossy compression, where a small amount of information is discarded to achieve much higher compression ratios.

For example, ZIP compression is lossless: after compression and decompression, every bit is identical to the original file. JPEG image compression, on the other hand, is lossy. The compressed image contains small visual artefacts that become more noticeable as the compression ratio increases. Once compressed, the discarded information cannot be recovered.

Lossless compression sounds ideal, but unfortunately it performs poorly on numerical time series. Sensor measurements and simulation outputs often resemble pseudo-random sequences from the compressor’s point of view, leaving very little redundancy to exploit. To obtain meaningful reductions in file size, some degree of lossy compression is generally unavoidable.

A first step: 32-bit floating-point values

The simplest solution is to store the data using 32-bit floating-point values instead of 64-bit doubles. This immediately halves the storage requirements, providing a 2:1 compression ratio.

For the vast majority of real-world datasets—sensor measurements, experimental recordings, or simulation results—the difference is practically invisible. The precision offered by single-precision floating-point numbers is already well beyond the accuracy of most measurement systems.

One could imagine reducing the precision further using 16-bit floating-point values, but these formats are less consistently supported across platforms and provide limited control over the available precision. Instead, D2xlab uses a different approach.

Integer block coding

To achieve significantly better compression while preserving as much information as possible, D2xlab uses an integer block coding scheme.

Each time series is divided into blocks of 32 consecutive samples.

For each block, the minimum and maximum values are determined. Every floating-point value in the block is then linearly mapped onto an integer scale:

• the smallest value becomes 0,
• the largest value becomes the largest representable integer,
• every other value is proportionally mapped in between.

For example, if a 16-bit integer representation is used, every block automatically makes full use of all 65,536 available levels, regardless of the signal’s absolute amplitude.

This differs fundamentally from floating-point representations, where much of the available precision may be wasted if the signal occupies only a small fraction of the representable range.

By adapting the scale independently for every block, D2xlab preserves the local dynamic range of the signal while requiring far fewer bits per sample.

The only additional information that needs to be stored for each block is its minimum and maximum values, allowing the original floating-point values to be reconstructed with a very small quantisation error.

In practice, this approach provides substantially better compression than simply converting everything to 32-bit floats, while introducing errors that are typically well below the intrinsic noise level of real-world sensors.

In the next section, we’ll quantify these errors on our washing machine dataset and see how different integer precisions affect both compression ratio and reconstruction accuracy.

Looking at the results

One interesting aspect of the washing machine recording is its wide dynamic range, with long quiet periods interrupted by sudden, high-amplitude vibrations. These changing signal levels make it a particularly challenging dataset for compression.

The figures below compare the reconstructed signals with the original recording. Both the time histories and the power spectral densities show that integer block coding preserves the signal remarkably well, even when using only an 8-bit integer representation.

Want to see the difference yourself? Open the washing machine dataset in D2xlab , then hold Ctrl while selecting all three datasets to compare the original recording with the 16-bit and 8-bit compressed versions.

Although the 8-bit representation introduces slightly more quantisation noise, the overall signal characteristics remain very well preserved. In many practical applications, even this level of compression would have little impact on the subsequent analysis.

The achieved compression ratios are summarised below:

Encoding	Compression ratio	Effective bits per sample
64-bit float	1:1	64
16-bit integer block coding	1:3.4	19
8-bit integer block coding	1:5.8	11

Note that there is a small overhead associated with integer block coding, since each block must also store its minimum and maximum values. As a result, the effective number of bits per sample is slightly higher than the nominal integer precision.

Conclusion

Most real-world sensors provide between 12 and 16 bits of effective resolution. Storing every sample as a 64-bit floating-point value is therefore often unnecessary once the numerical computations have been completed.

By combining block-wise normalisation with 16-bit integer quantisation, D2xlab achieves compression ratios of more than 3:1 while preserving virtually all of the useful information contained in the original recording.

This integer block coding is the format D2xlab uses whenever you share a dataset. It reduces upload times and storage requirements while preserving the information needed for virtually all subsequent analyses.