EOSC Synergy Storage solutions and access
Storage is required for all applications and services, and exists in various different ways and protocols. Here we present a set of solutions that are actively used in the EOSC Synergy project. The solution you choose might be limited to the selected cloud solution and available technologies in your VO.
With this section, we present a list of tools used by services tested or already running at EOSC Synergy. This list include free and comercial storage options with different knowledge requirements and difficulty depending on the experience of the developer.
- Fedcloud Openstack dashboard: dashboard.fedcloud.eosc-synergy.eu
- Online tutorial : Tutorial-OpenStack-Dashboard
Another solution by OpenStack. If you need fast data access with infinite scalability (no need to reshape volumes) you probably should look into the into Object Storage technology. Swift (OpenStack) is the storage alternative to Cinder and probably one of your better options to work with object storage.
There are multiple ways to access Swift storage, however they might not be so intuitive. If you decide to use rclone to synchronise your filesystem with Switf, the tool EGI Swift Finder can implement all the discovery and configuration for you.
Documentation, examples and utils can be found at:
- EGI Swift Finder: github.com/lburgey/egiSwiftFinder
- rclone example: rclone.org/swift
Documentation and examples can be found at:
- User manual: rclone.org/docs
- Swift example: rclone.org/swift
- WebDAV example: rclone.org/webdav
Nextcoud is designed to access via the web interface and WebDAV. Authentication via EGI Check In currently only works for the web interface. To use WebDAV and other protocols, currently passwords or OAuth2 Tokens have to be created in the web interface, before they can be used on the commandline.
Documentation, examples and the repository can be found at:
- Admin manual: docs.nextcloud.com/server/latest/admin_manual
- User manual: docs.nextcloud.com/server/latest/user_manual
- Accessing Nextcloud files: docs.nextcloud.com/server/latest/user_manual/en/files
- OIDC connect user backend: github.com/nextcloud/user_oidc
- Additional authentication extensions: github.com/nextcloud/user_external
Network File System (NFS)
You can also deploy the NFS server in one of your Federated Cloud VMs to share data with other VMs on the same site.
The down side is that NFS does not offer a big variety of authentication and access methods. Security of all data in the NFS share depends on the integrity of all clients and the security of the network connections. Therefore, NFS is available inside Computer Centres only, and not exported.
Documentation and interesting use cases can be found at:
- Kubernetes volume example: github.com/kubernetes/examples/tree/master/staging/volumes/nfs
- RCF document: datatracker.ietf.org/doc/html/rfc7530
With Infrastructure Manager, deployment is just a matter of few clicks. There are multiple deployment choices and probably one of them is a server running your favourite database.
Although, if unfortunatelly you need something special or there is not an instace for your case yet, you can choose "Start a virtual machine with extra HD" and just run your database service there (similaryl to Nextcloud). Remember, if you deploy your database application this way, probably you want to check the server and firewall ports at your VM and OpenStack.
Relevan databases and generic infrastructure options available at:
- Infrastructure Manager Dashboard: appsgrycap.i3m.upv.es:31443/im-dashboard
- MongoDB docker image: hub.docker.com/_/mongo
- PostgreSQL docker image: hub.docker.com/_/postgres
A data management solution trying to provide High-performance with unified data access across globally distributed environments. If you have a very distributed cluster that your services need to access, this is probably an option for you.
The data organisation and sharing is similar to a filesystem, users organise their data in virtual volumes called spaces and shares access between groups. To access your data you have multiple options such web interface, CLI (command-line interface) or an API. Authentication and authorisation are based OpenID Connect and SAML, supporting as well the usage of tokens at API level.
Documentation is available at:
- Official documentation: onedata.org/#/home/documentation
B2SHARE is a user-friendly, reliable and trustworthy way for researchers, scientific communities and citizen scientists to store, publish and share research data in a FAIR way. B2SHARE is a solution that facilitates research data storage, guarantees long-term persistence of data and allows data, results or ideas to be shared worldwide. B2SHARE supports community domains with metadata extensions, access rules and publishing workflows. EUDAT offers communities and organisations customised instances and/or access to repositories supporting large datasets.
To manage your data thre are a web interface and HTTP API. Authentication and authorisation are based on password or OIDC, using access tokens in the case of the API. Note that EUDAT encourages FAIR principles, so double check the privacy of your data (e.g. Metadata is always publicly available).EUDAT introduction and demos:
More information and examples can be found at :
- User documentation: eudat.eu/services/userdoc/b2share-usage
- B2SHARE HTTP REST API documentation: eudat.eu/services/userdoc/b2share-http-rest-api