Nextcloud’s “Simple Sign-up” offers the possibility to get a free account with any Nextcloud provider. “Free services are of course paid somewhere by someone. The listed Nextcloud providers all give a free account with 2 to 5 GB storage but offer larger storage and more options like a dedicated setup for a fee, which funds the free accounts. Consider paying for your account, it helps them provide this service!” (Quote https://nextcloud.com/de/signup/)
In the spirit of open source, we have been running Simple Sign-up since November 2020 on cloud3.linuxfabrik.io. This free service with 5 GB of storage comes with no SLA, no warranty and no support. To date, 3,000 users have opted for this service.
The user files of the free cloud offering are stored on a dedicated, intrinsically redundant storage system from a well-known manufacturer, and deliberately so: for cost and security reasons, the offering is not linked to our productive storage environment.
TL;DR: What happened?
Users reported repeated outages starting on Saturday, 11/13/2021. From Sunday, 11/14/2021 to Friday, 11/19/2021 cloud3.linuxfabrik.io was down: the storage system used for the service was broken after an automatic update and the files were gone. The database of the service is running somewhere else and is not affected, so data like contact and calendar entries are still available.
How bad is it really?
If you work with a sync client, you will not notice any data loss except for the downtime – the files are still on your own computer. Contacts, calendar entries, bookmarks, etc. are available again as usual.
Was only Nextcloud Simple Sign-up affected?
Yes. The disruption only affected this one free service. Our commercial offerings were not affected because they run on a completely different infrastructure and open source-based storage systems.
Selected reactions of some users, uncommented:
- I’M TRYING TO SYNC MY FILES! PLEASE FIX IT!!
- PLEASE FIX IT! I NEED TO ACCESS MY FILES!!
- your free storage has been broken for 2 days, I would like to know if I can hope to get my data back and I just need to be patient or I just have to move on because it is lost forever ?
- wann wird der Defekt behoben werden?
- You should have definitely shared more updates on Twitter.
- Warum ist das Problem nur den Kostenfreien Diensten aufgetreten? Habt Ihr kein Backup?????? [gekürzt] Wie stellt Ihr sicher, dass es keine Probleme beim Produkt “nextcloud-enterprise” gibt, auch da habt Ihr kein Backup? Über diesen Vorfall, kein Backup bin ich richtig baff.
- Wichtig sind mir jedoch meine Kalendereinträge und Kontaktdaten. Ich benötige davon einen Export. Können Sie diesen anstoßen und mir senden, oder den Zugriff darauf zulassen? Oder betrifft es auch diese Daten? Das wäre grauenvoll.
- Hallo, meine Fragen sind nicht beantwortet.
- I was experiencing major downtime on your free cloud3.linuxfabkik.io, and I want to retrieve my data from you, make a backup if you like. How can I do it?
- Nicht euer Ernst oder? Bin deswegen ziemlich angesäuert. Ich weiss, ich nutze oder besser gesagt nutzte nur das “Gratisprodukt”. Aber sorry, dass darf nicht passieren. Auf eurem Produkt steht CH und nicht irgendwie Togo oder so etwas ähnliches….
Dubious content providers had discovered our Nextcloud offering for themselves and used an account to distribute their content via public Nextcloud shares. Early users reported that the service was down every now and then. The reason was overload, the allocated network bandwidth was exhausted. After we deleted the causing user, the problem was solved.
Regardless of the first incident, the storage system automatically applied an update, which was unfortunately withdrawn by the manufacturer a few hours later. We had deactivated automatic updates during configuration, but apparently they were reactivated after one of the previous updates. How, cannot be reconstructed anymore. Afterwards, all volumes on this system were logically lost, even if the disks still showed an allocation. All attempts to remount the volumes failed.
We reported about the outage on Twitter and prepared a mailing list. At the same time, we continuously drew the manufacturer’s attention to the problem, but support was slow to come up with useful answers. Our attempts to access the volumes with various tricks were unsuccessful – it became more and more likely that the files stored on them were lost. Several times we set a deadline for the manufacturer to respond to our support case. We could not and did not want to wait any longer.
It came as it had to: still no useful answer. Even though this annoys us massively, we do not do fingerpointing and do not name the manufacturer of the storage system. We consider the files lost at the moment and re-enable the service to the outside world with an empty file base and no upload capability. So file synchronization does not work, but all other data like calendar, contacts and bookmarks can be downloaded.
Of course, this should not happen – but unfortunately it did. The service is currently available without the storage system. We’ll stay tuned and see what happens next.
No response from manufacturer support. We took another look at the storage system ourselves with fresh ideas, but unfortunately without success in the end.
Still no answer from the manufacturer’s support. Our deadline has thus expired, the attached storage is no longer salvageable from our point of view.
We will therefore shut down the free Nextcloud Simple Sign-up environment on 15.12.2021. Please backup your remaining data (calendar, contacts, bookmarks etc.) until then. Who likes, can delete his account already before 15.12. itself. If you still need a free Nextcloud account, we recommend you to use another provider.