> when someone opens up the hard drive of a redbox unit, they can pull a file which has a complete list of titles ever rented, and the email addresses of the people who rented them, and where and when
ticoombs
1w ago
•
100%
Article says the initial compromise of the non-airgapped systems is an unknown vector. So how they got into the organisation(s) in the first place is still a mystery
ticoombs
1w ago
•
100%
Turns out it wasn't the certificate, but the post-renewal restart of the web-server wasn't working.
We've fixed that up and now LW content is coming in hot and fast!
Our proxy in Amsterdam has a reddthat.com SSL certificate. It didn't renew correctly. We are investigating
ticoombs
1mo ago
•
100%
This is sso support as the client. So you could use any backend that supports the oauth backend (I assume, didn't look at it yet).
So you could use a forgejo instance, immediately making your git hosting instance a social platform, if you wanted.
Or use something as self hostable like hydra.
Or you can use the social platforms that already exist such as Google or Microsoft. Allowing faster onboarding to joining the fediverse. While allowing the issues that come with user creation to be passed onto a bigger player who already does verification. All of these features are up for your instance to decide on.
The best part, if you don't agree with what your instance decides on, you can migrate to one that has a policy that coincides with your values.
Hope that gives you an idea behind why this feature is warranted.
ticoombs
1mo ago
•
100%
Possibly, as it's one generic endpoint, but it also blocked a few other things people in the fediverse created, which are mighty helpful in diagnosis of these and other issues.
So using some AI model or whatever CF uses is probably not going to be the best thing for us as it classified a POST request as a crawler?? 🤷
I'd have to whitelist every regular endpoint as well and then it gets messy as CF only gives you so much control as a free user.
So, for the moment I've blocked the most annoying ones based on UserAgent.
ticoombs
1mo ago
•
90%
We enabled the CloudFlare AI bots and Crawlers mode around 0:00 UTC (20/Sept).
This was because we had a huge number of AI scrapers that were attempting to scan the whole lemmyverse.
It successfully blocked them... While also blocking federation 😴
I've disabled the block. Within the next hour we should see federation traffic come through.
Sorry for the unfortunate delay in new posts!
Tiff
to be paired with [tang](https://github.com/latchset/tang)
ticoombs
1mo ago
•
100%
Thankyou! That made my day! (And the servers filled with coffee)
Not so much a sploit but an easy way to do broadcasting!
We had a brief outage today due to the server running out of space. I have been tracking our usage but associated it with extra logging and the extra build caches/etc that we've being doing. Turns out the problem was the frontend Next-UI which has been caching every image since the container was created! All 75GB of cached data! Once diagnosed it was a simple solution to fix. I'm yet to notify the project of this error/oversight and I'll edit this once Issues/PRs are created. I also haven't looked at turning the caching off yet as my priority was recovering the main Reddthat service. Thanks all for being here! Tiff
ticoombs
2mo ago
•
100%
I can neither confirm or deny for the safety of my pigeons.
ticoombs
2mo ago
•
100%
I've always wanted to do colocation and looked into it when I first started Reddthat and we had our initial growth. We are lucky we didn't otherwise I would be out of money ages ago!
The aruba.it colocation is about the same price as our OVH server which would be probably the most viable as we are close to having enough donators to have a long enough runway it would make sense.
(Goal being: A$150/m ~ USD$100/€90. And OpenCollective doing some magical nonsense with the Total amount we have left and averaging it across 12 months or something)
The 32GB of RAM we have currently is overkill for an instance of our size. We could get away with using 16GB and still have room to grow for the next year. Which will probably be what will happen in April, but I'll re-evaluate before then and see whether a colocation option is viable.
Our stance on downvotes, and lack of super-popular local communities means we are growing slowly over time and are very stable in requirements. Lemmy still being in it's infancy means there are problems that are no fault of the hardware but the software. An example would be the latest issue we were experiencing where no matter what resources we had, it would result in 100% CPU usage for brief periods, with 10-20s page loads for everyone during that period.
I am very tempted but I still need to make sure I am being responsible with the donations. By next year we'll most likely be over 1TB of object storage and our S3 costs will slowly start increasing MoM. While not a lot I still need to factor it in growing costs.
Recently I've taken the docker compose example from SChernykh and have started a p2pool for Reddthat! https://github.com/SChernykh/p2pool/tree/master/docker-compose (many thanks here!). After some minor changes I removed the IP listing from statistics and increase the visibility to 100 "supporters". It's viewable at donate.reddthat.com. (if @admin@monero.town wants the code change I can provide diff) The idea was to also allow people to donate to instances via CPU instead of actual $. My question for the community is whether I am creating a centralised pool or am I still participating in a decentralized fashion?
ticoombs
2mo ago
•
100%
Yeah! Except it wasn't just images. Instead it was all links. And inside the Lemmy process and it would block all other connections! Which is the why it was an issue! Super weird.
But the whole Lemmy app is full of these issues hiding in the background. People seem to forget it's still in its infancy and expect a 1-1 Reddit experience. We regularly get people signing up and then quitting like the thread from last week regarding the lack of downvotes. Or the lack of X,Y,Z.
</end rant>
It's fixed for Reddthat now, which is the best part! 😉
ticoombs
2mo ago
•
100%
It's live!
ticoombs
2mo ago
•
50%
I"M A MOD LOOK AT ME
ticoombs
2mo ago
•
50%
replying
ticoombs
2mo ago
•
50%
commenting
here's a graph showing when we did the deploy!
ticoombs
2mo ago
•
100%
I do! It's already in the sidebar!
BTC Directly: bc1q8md5gfdr55rn9zh3a30n6vtlktxgg5w4stvvas
ticoombs
2mo ago
•
100%
😍 Thank you for being here!
ticoombs
2mo ago
•
100%
❤️ the heart emoji doesn't do it justice! Thankyou!
ticoombs
2mo ago
•
100%
🤩 Always great to hear! Thanks!
ticoombs
2mo ago
•
100%
The long loads are because of huge images/content believe it or not 😂. I too can't wait to finally see some nice fast Reddthat
ticoombs
2mo ago
•
100%
Whoa! Thanks for even considering donating. I won't hold you to it if you happen to donate less later 😉
After testing ko-fi we still end up having the same fees compared with OpenCollective as it's PayPal instead of Stripe. So in the end it's better to go via OpenCollective. As it's a lot more transparent and shows all the donations and will allow me to show all of our bills etc.
Thanks!
Hello. It is I, Tiff. I am not dead contrary to my lack of Reddthat updates 😅 ! It's been a fun few months since our last update. We've been mainlining those beta releases, found critical performance issues before they made it into the wider Lemmyverse and helped the rest of the Lemmyverse update from Postgres 15 to 16 as part of the updates for Lemmy versions 0.19.4 and 0.19.5! Thank-you to everyone who helped out in the matrix admin rooms as well as others who have made improvements which will allow us to streamline the setup for all [future upgrades](https://github.com/LemmyNet/lemmy/pull/4892). And a huge thank you to everyone who has stuck around as a Reddthat user too! Without you all this little corner of the world wouldn't have been possible. I havn't been as active as I should be for Reddthat, moderating, diagnosing issues and helping other admins has been taking the majority of my Reddthat allocated time. Creating these "monthly" updates should... be monthly at least! so I'll attempt start posting them monthly, even if nothing is really happening! # High CPU Usages / Long Load Times Unfortunately you may have noticed some longer page load times with Reddthat, but we are not alone! These issues are with Lemmy as a whole! Since the 0.19.x releases many people have talked about Lemmy having an increase in CPU usage, and they have the monitoring to prove it too. On average there was a 20% increase and for those who have single user instance this was a significant increase. Especially when people were using a raspberry pi or some other small form factor device to run their instance. This increase was one of the many reasons why our server migrations were required a couple months ago. There is good news believe it or not! We found the issue with the long page load times, and helped the developers find the [solution](https://github.com/LemmyNet/lemmy/pull/4957)! - This change looks like it will be merged within the next couple days. Once we've done our own testing, we will backport the commit and start creating our own Lemmy 'version'. Any backporting will be met with scrutiny and I will only cherry-pick the bare minimum to ensure we never get into a situation where we are required to use the beta releases. Stability is one of my core beliefs and ensuring we all have a great time! # Donation Drive We need some recurring donations!  We currently have $374.10 and our operating costs have slowly been creeping up over the course of the last few months. Especially with the current currency conversions. The current deficit is $74. Even with the amazing 12 current users we will run out of money in 5 months. That's January next year! We need another 15 users to donate $5/month and we'd be breaking even. That's 1 coffee a month. If you are financially able please see the sidebar for donation options, go to our [funding post](https://reddthat.com/post/25633) , or go directly to our [Open Collective](https://opencollective.com/reddthat) and signup for recurring donations! Our finances are viewable to all and you can see the latest expense here: https://opencollective.com/reddthat/expenses/213722 - OVH Server (Main Server) - $119.42 AUD - Wasabi S3 (Image Hosting) - $16.85 AUD - Scaleway Server (LemmyWorld Proxy) - $6.62 AUD ## Scaleway Unfortunately until Lemmy optimises their activity sending we still need a proxy in EU, and I havn't found any server that is cheaper than €3.99. If you know of something with 1GB RAM with an IPv4 thats less than that let me know. The good news is that Lemmy.ml is currently testing their new sending capabilities so it's possible that we will be able to eventually remove the server in the next year or so. The biggest cost in scaleway is actually the IPv4. The server itself is less than €1.50 so if lemmy.world had IPv6 we could in theory save €1.50/m. In saying all this, that saving per month is not a lot of money! ## Wasabi Wasabi S3 is also one of those interesting items where in theory it should only be USD$7, but in reality they are charging us closer to USD$11. They charge a premium for any storage that is deleted before 30 days, as they are meant to be an "archive" instead of a hot storage system. This means that all images that are deleted before 30 days incur a cost. Over the last 30 days that has amounted to 305GB! So while we don't get charged for outbound traffic, we are still paying more than the USD$7 per month. We've already tried setting the pictrs caching to auto-delete the thumbnails after 30 days rather than the default 7 days, but people still upload and delete files, and close our their accounts and delete everything. I expect this to happen and want people to be able to delete their content if they wish. ## OVH Server When I migrated the server in April we were having database issues. As such we purchased a server with more memory (ram) than the size of the database, which is the general idea when sizing a database. `Memory: 32 GB`. Unfortunately I was thinking on a purely technical level rather than what we could realistically afford and now we are paying the price. Literally. (I also forgot it was in USD not AUD :| ) Again, having the extra ram gives us the ability to host our other frontends, trial new features, and ensure we are going to be online incase there are other issues. Eventually we will also increase our Lemmy applications from 1 to 2 and this extra headroom will facilitate this. # Donate your CPU! (Trialing) If you are unable to donate money that is okay too and here is another option! You can donate your CPU power instead to help us mine crypto coins, anonymously! This is a new initiative we are trialing. We have setup a P2Pool at: https://donate.reddthat.com. More information about joining the mining pool can be found there. The general idea is: download a mining program, enter in our pool details, start mining, when our pool finds a "block", we'll get paid. I've been testing this myself as an option of the past month as a "side hustle" on some laptops. Over the past 30 days I managed to make $5. Which is not terrible if we can scale it out. If it doesn't takes off, that's fine too! I understand some people will be hesitant for any of the many reasons that exist against crypto, but any option to help us pay our server bills and allow people to donate in an anonymous way is always a boon for me. # Conclusion These Lemmy bugs have been causing a headache for me in diagnosing and finding solutions. With the upcoming 0.19.6 release I hope that we can put this behind us and Reddthat will be better for it. Again, thank you all for sticking around in our times of instabilities, posting, commenting and engaging in thoughtful communications. You are all amazing. Cheers, Tiff
the "official" incident root cause analysis
arstechnica.com
An article from July, but I bet you haven't updated your bios! Or you left it open on purpose?
ticoombs
3mo ago
•
100%
🤷