• Storage Access

    EOSC Synergy Storage solutions and access

    Storage is required for all applications and services, and exists in various different ways and protocols. Here we present a set of solutions that are actively used in the EOSC Synergy project. The solution you choose might be limited to the selected cloud solution and available technologies in your VO.

    With this section, we present a list of tools used by services tested or already running at EOSC Synergy. This list include free and comercial storage options with different knowledge requirements and difficulty depending on the experience of the developer.


    OpenStack Cinder

    If your cloud is hosted on an infrastructure managed by OpenStack, this type of storage will be the easiest you can access and use. It is available via OpenStack dashboard and will look like just like a harddrive in your VM.
    However, note with this solution only the users and services with access to your VM can access the storage folder.
    If you need to authorised access from community users but you do not want to provide VM access, you probably have to look for another alternative. However, you can still use Cinder to extend your VM storage or combine it with to other solutions providing the interface you like the most (e.g. Nextcloud).

    Documentation, examples and the repository can be found at:


    OpenStack Swift

    Another solution by OpenStack. If you need fast data access with infinite scalability (no need to reshape volumes) you probably should look  into the into Object Storage technology. Swift (OpenStack) is the storage alternative to Cinder and probably one of your better options to work with object storage.

    There are multiple ways to access Swift storage, however they might not be so intuitive. If you decide to use rclone to synchronise your filesystem with Switf, the tool EGI Swift Finder can implement all the discovery and configuration for you. 

    Documentation, examples and utils can be found at:


    RClone

    Rclone is a command line program to manage files on cloud storage. It is a feature rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.

    Rclone mounts any local, cloud or virtual filesystem as a disk on Windows, macOS, linux and FreeBSD, and also serves these over SFTP, HTTP, WebDAV, FTP and DLNA. Authentication and Authorisation will depend on the protocol you choose.

    Documentation and examples can be found at:


    Nextcloud


    Nextcloud is a suite of client-server software for creating and using file hosting services. Software is free and open-source making anyone allowed to install and operate it on their own private server devices.  Mange and access your files knowing your data is in your data center, on a server managed by you or your team, rather than floating somewhere in the cloud. It is simple to install and deploy, for example in one of your hosts at the Federated Cloud. 

    Nextcoud is designed to access via the web interface and WebDAV. Authentication via EGI Check In currently only works for the web interface. To use WebDAV and other protocols, currently passwords or OAuth2 Tokens have to be created in the web interface, before they can be used on the commandline. 

    Documentation, examples and the repository can be found at:


    Network File System (NFS)

    Old but gold. Developed originally at 1984, NFS is an open IETF standard with software available for almost all Operative Systems. Some of the Synergy Sites provide a local NFS server, that allows mounting it into your Virtual Machines.

    You can also deploy the NFS server in one of your Federated Cloud VMs to share data with other VMs on the same site.

    The down side is that NFS does not offer a big variety of authentication and access methods. Security of all data in the NFS share depends on the integrity of all clients and the security of the network connections. Therefore, NFS is available inside Computer Centres only, and not exported.

    Documentation and interesting use cases can be found at:


    Database Server

    With Infrastructure Manager, deployment is just a matter of few clicks. There are multiple deployment choices and probably one of them is a server running your favourite database.

    Although, if unfortunatelly you need something special or there is not an instace for your case yet, you can choose "Start a virtual machine with extra HD" and just run your database service there (similaryl to Nextcloud). Remember, if you deploy your database application this way, probably you want to check the server and firewall ports at your VM and OpenStack.

    Relevan databases and generic infrastructure options available at:


    OneData

    A data management solution trying to provide High-performance with unified data access across globally distributed environments. If you have a very distributed cluster that your services need to access, this is probably an  option for you.

    The data organisation and sharing is similar to a filesystem, users organise their data in virtual volumes called spaces and shares access between groups. To access your data you have multiple options such web interface, CLI (command-line interface) or an API. Authentication and authorisation are based OpenID Connect and SAML, supporting as well the usage of tokens at API level.

    Documentation is available at:


    B2Share

    B2SHARE is a user-friendly, reliable and trustworthy way for researchers, scientific communities and citizen scientists to store, publish and share research data in a FAIR way. B2SHARE is a solution that facilitates research data storage, guarantees long-term persistence of data and allows data, results or ideas to be shared worldwide. B2SHARE supports community domains with metadata extensions, access rules and publishing workflows. EUDAT offers communities and organisations customised instances and/or access to repositories supporting large datasets.

    To manage your data thre are a web interface and HTTP API. Authentication and authorisation are based on password or OIDC, using access tokens in the case of the API. Note that EUDAT encourages FAIR principles, so double check the privacy of your data (e.g. Metadata is always publicly available).

    EUDAT introduction and demos:


    More information and examples can be found at :


    ◄Federated CloudAuthentication Tools►