sys-cluster/sionlib/metadata.xml


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE pkgmetadata SYSTEM "https://www.gentoo.org/dtd/metadata.dtd">
<pkgmetadata>
	<maintainer type="person">
		<email>lssndrbarbieri@gmail.com</email>
		<name>Alessandro Barbieri</name>
	</maintainer>
	<longdescription lang="en">
SIONlib is a library for writing and reading data from several thousands of parallel tasks into/from one or a small number of physical files. Only the open and close functions are collective while file access can be performed independently.

SIONlib can be used as a replacement for standard I/O APIs (e.g. POSIX) that are used to access distinct files from every parallel process. SIONlib will bundle the data into one or few files in a coordinated fashion in order to sidestep sequentialising mechanism in the file system. At the same time, the task-per-file picture is maintained for the application, every process has access to its logical file only. File access is performed using SIONlib equivalents to standard C-I/O functionality (fwrite becomes sion_write, fseek becomes sion_seek, etc.) which have similar semantics as their C counterparts.

Internally, the physical files are sub-divided into sequences of blocks, which themselves contain one chunk of data belonging to every logical file. In case the amount to be written to a file is known up front, it can optionally be specified when opening the file and the sequence of blocks collapses into a single block with one chunk per task containing all of its data. If a chunk size cannot be specified ahead of time, a sensible default is chosen and reads and writes that cross chunk boundaries are handled transparently by SIONlib.

SIONlib also uses information about the block size of the underlying file system, because access to the same block from different tasks often leads to contention.

Both, the estimated chunk size and file system block size are used to align individual chunks with file system blocks. Ensuring contention-free access to file system blocks enables efficient parallel writing and reading.

SIONlib provides two different interfaces: one for parallel access (with implementations for different parallel programming technologies such as MPI, OpenMP and hybrid MPI+OpenMP) and one for sequential access which is also used internally by the SIONlib utilities.
	</longdescription>
	<upstream>
		<bugs-to>https://gitlab.jsc.fz-juelich.de/cstao-public/SIONlib/SIONlib/-/issues</bugs-to>
	</upstream>
	<use>
		<!--<flag name="cuda">enable CUDA aware interface</flag>-->
		<flag name="cxx">Enable C++ support</flag>
		<flag name="debug">enable SIONlib debug</flag>
		<flag name="deep-est-sdv">deep-est-sdv collector selection plug-in for MSA-aware collective I/O</flag>
		<flag name="fortran">Enable Fortran support</flag>
		<flag name="hostname-regex">hostname-regex collector selection plug-in for MSA-aware collective I/O</flag>
		<flag name="mpi">enable mpi</flag>
		<flag name="python">enable python support</flag>
		<flag name="ompi">enable hybrid openmp mpi</flag>
		<flag name="openmp">enable openmp</flag>
		<flag name="parutils">Enable compilation of parutils (used for benchmarking)</flag>
		<flag name="pthreads">configure SIONlib to use pthreads for locking</flag>
		<flag name="sionfwd">enable I/O forwarding with <pkg>sys-cluster/SIONfwd</pkg></flag>
	</use>
</pkgmetadata>