There's a not-entirely-small part of me that hopes Issues breaks entirely for long enough that people at GitHub rethink their implementation.
GH issues have been so useful for the better part of a decade, that I have an empty repo called my_life just so I can make issues about things like home maintenance. In the last few months, the UI has become so slow and flaky that I don't use it for nearly as much as I used to. And when I have to for a project, it slows me down noticeably.
I thought I was alone in this, but asking around I found that this is a common frustration among developers I communicate regularly with.
I do the same - I've been running so many non-software things out of private GitHub Issues. I even have a simple system for creating a new issue every day to use as a personal TODO list, described here: https://til.simonwillison.net/github-actions/daily-planner
I've noticed everything feeling a whole lot less snappy and responsive over the past few days - ironically I think it's because they've done a major rewrite of the frontend presumably with the aim of making it more snappy and responsive!
I’m low-key certain it’s because they started using more React on the front end, especially for the new Issue stuff. The entire UI feels slow and janky like GitLab.
You're not alone. There's been an effort to transparently update the UI to a React implementation over the past year or two, and while I understand the benefits to that approach, they have introduced some flakiness in moving away from a the server-rendered pjax/html-pipeline/simple web components approach that was so cohesive and battle tested over the decade before it.
That's actually a really interesting way to leverage that feature. Have you found this easier than other services built specifically for this use case?
I've tried other tools. But I'm on GitHub most days, so it's been a seamless way to keep track of some things that would otherwise disappear into a calendar, or some tool that I don't use as often.
Seconded. Hosting FOSS on a platform that isn't FOSS itself and run by a for-profit company who tried to work against FOSS for so long (Microsoft) doesn't make much sense to me.
Instead, we should dogfood the FOSS ecosystem on a platform that is FOSS itself, and run as a non-profit. Codeberg, for better or worse, is the best platform for this today.
Personal projects are on a vps and accessed through ssh. I’ve self-hosted gitea, then forgejo, then found out that I don’t like the interface or the auth dance. My plan is to ise cgit if I want something to be public and any forge if the purpose is collaboration (sourcehut is nice)
"are you sure your self-hosted solution will have better uptime?"
Hilariously, after 15 years working in self hosted bitbucket systems, YES entirely.
An underfunded university with an incompetent but doing their best IT department? Zero downtime.
A mid sized company full of overly confident and "just build it" nerds building fragile shit? Zero downtime.
A large corporation with a completely outsourced IT department that can't give you access to something unless you do exactly the right undocumented thing in our internal ticketing software? Zero downtime.
That includes self hosted jenkins and literally homebuilt infrastructure with zero documentation and the guy that built it left a while ago.
I have never been able to blame our build and code infra for lack of productivity.
I'm seeing issues with repo pages entirely (getting Unicorn "We couldn't respond to your request in time" errors) but the status page hasn't updated to show that yet.
Besides projects/companies/people stuck on Bitbucket, is anyone actually voluntary using Bitbucket? I remember using it back when I was poor and it was the only choice for hosted free private repositories, but the rest of the platform was actively worse in basically every way compared to the alternatives.
This continues to be a regular thing for the past 5 years of outages and I've lost count of the many outages that GitHub continues to have since calling it. [0]
They're saying PRs and issues but I can't access github.com or github oauth for logging into external services. Need to migrate our org off of using github auth for anything critical I think, this is twice so far this year that we've had these issues.
Or, GitHub used to favor stability back in the day, and there wasn't a lot of changes. People were complaining that GitHub didn't "improve" enough day-to-day, so after the Microsoft purchase, Microsoft started forcing GitHub to add more features, stability be damned.
Most outages are caused not by stuff just randomly breaking, but updates/upgrades going wrong. If you try to increase the output of new features/changes to a platform, you're bound to have more outages and downtime if you aren't more careful than before.
Microsoft, who never really excelled at engineering, to the surprise of absolutely everyone, choose adding features over stability and since years back, we're seeing the consequences of that choice.
I'm not totally certain you can attribute it to this. I know there's a bunch of work going on to migrate things into Azure after the MS acquisition but it feels like more of an industry trend that we cut engineering spend at the cost (often) of lower quality output and outages like these.
GitHub laid off 10% of their staff in 2023 and like you say, won't have slowed down to account for that.
Ignoring all that though... other than Copilot, what big feature changes have you seen in GitHub? My experience of using their product has been broadly unchanged for years.
> My experience of using their product has been broadly unchanged for years.
Compared to how fast/slow they were moving 2012-2018, they're moving at blazing speed now. It feels like every time I open GitHub now, there are new features/changes and "Beta" available stuff. The platform is almost completely different today than it was in 2018, for better or worse.
I wouldn't say this applies to cloud providers, they have a very different business on their hands.
But for SaaS in general I think the trend is noticeable? Twitter led the way with massive layoffs in engineering often in the roles around reliability. The industry as a whole have aimed to cut costs however possible, and reliability/ops is usually seen as a cost-centre that gets hit hard.
I've watched this in my own space (start/scale-ups and larger companies, I work in incident response tooling) as people start talking very differently about reliability and engineering investment. You hear "do more with less" about five times every day and spend that was previously greenlit by default around reliability/redundancy is under much more scrutiny now.
I see this as a silent mirror of the reduction in open-source efforts from companies now there's been a refocus on business impact and bottom-line.
The last few days, the “create pr” screen kept hanging for me. I figured out I could reload it, click “create”, grab a coffee, and then come back and it’d be at the next screen.
Maybe it’s time to learn their CLI (and then patiently wait for work to switch to another git service).
One thing I started doing a few years ago is that, any time I check out a git repo, whether it's on github or anywhere else, is clone it to my NAS and then clone from there to whatever I'm building it on. Aside from guarding against losing dependencies to outages (or from being yanked), it can be handy for guarding against my own fat fingers, and for making sure that the same version is running on multiple systems.
I honestly can't believe how unreliable Github is. Outages are commonplace. It boggles my mind how nothing has been done to address the reliability regressions that have been creeping in ever since MS took over.
In a universe, where Obi-Wan Kenobi is a developer: "I felt a great disturbance in the workflow, as if millions of requests suddenly cried out in terror and were suddenly silenced. I fear something terrible has happened."
There's a not-entirely-small part of me that hopes Issues breaks entirely for long enough that people at GitHub rethink their implementation.
GH issues have been so useful for the better part of a decade, that I have an empty repo called my_life just so I can make issues about things like home maintenance. In the last few months, the UI has become so slow and flaky that I don't use it for nearly as much as I used to. And when I have to for a project, it slows me down noticeably.
I thought I was alone in this, but asking around I found that this is a common frustration among developers I communicate regularly with.
I do the same - I've been running so many non-software things out of private GitHub Issues. I even have a simple system for creating a new issue every day to use as a personal TODO list, described here: https://til.simonwillison.net/github-actions/daily-planner
I've noticed everything feeling a whole lot less snappy and responsive over the past few days - ironically I think it's because they've done a major rewrite of the frontend presumably with the aim of making it more snappy and responsive!
I’m low-key certain it’s because they started using more React on the front end, especially for the new Issue stuff. The entire UI feels slow and janky like GitLab.
You're not alone. There's been an effort to transparently update the UI to a React implementation over the past year or two, and while I understand the benefits to that approach, they have introduced some flakiness in moving away from a the server-rendered pjax/html-pipeline/simple web components approach that was so cohesive and battle tested over the decade before it.
I am not experiencing a full outage right now, but I am experiencing very slow (spinner) and flakey (error that works again on repeat).
So perhaps it's the same problem, that's been getting more prevalent until it was noticeable by monitoring. One can hope!
That's actually a really interesting way to leverage that feature. Have you found this easier than other services built specifically for this use case?
I've tried other tools. But I'm on GitHub most days, so it's been a seamless way to keep track of some things that would otherwise disappear into a calendar, or some tool that I don't use as often.
For something so critical, does it make sense to self host say a gitea instance or such?
I know you can sync repos, I don't think you can sync issues tho
this is what i doits worth the 11 dollars a year for the vps to control my own
Who's going to kick off the holy war around self hosting?
Speaking of self hosting, Codeberg [1] is great. It's open and it looks and behaves pretty much like GitHub (unlike GitLab).
[1] https://codeberg.org/
Seconded. Hosting FOSS on a platform that isn't FOSS itself and run by a for-profit company who tried to work against FOSS for so long (Microsoft) doesn't make much sense to me.
Instead, we should dogfood the FOSS ecosystem on a platform that is FOSS itself, and run as a non-profit. Codeberg, for better or worse, is the best platform for this today.
Did you mean to say Forgejo? Correct me if I'm wrong, but Codeberg is simply hosting a Forgejo instance for you, so not exactly self hosting.
Since you asked, https://about.gitea.com/ is a great tool. MIT license.
Would be even better if they didn't require a CLA to contribute.
Personal projects are on a vps and accessed through ssh. I’ve self-hosted gitea, then forgejo, then found out that I don’t like the interface or the auth dance. My plan is to ise cgit if I want something to be public and any forge if the purpose is collaboration (sourcehut is nice)
Honestly, sounds like a lot of work and I'm not sure I get much benefit, or even less downtime.
I don't know about others but for me github is hardly a service that I'm sweating moment to moment uptime. My apps are still running and so on.
Hosting GitLab is not fun but anyways, M$ can still suck it.
Not Onedev or Forgejo, though it could get tricky depending on what features you require.
It all began 5 years ago [0] when we wanted to 'centralize everything to GitHub.'
Then, the outages accelerated from there.
[0] https://news.ycombinator.com/item?id=22868406
"are you sure your self-hosted solution will have better uptime?"
Hilariously, after 15 years working in self hosted bitbucket systems, YES entirely.
An underfunded university with an incompetent but doing their best IT department? Zero downtime.
A mid sized company full of overly confident and "just build it" nerds building fragile shit? Zero downtime.
A large corporation with a completely outsourced IT department that can't give you access to something unless you do exactly the right undocumented thing in our internal ticketing software? Zero downtime.
That includes self hosted jenkins and literally homebuilt infrastructure with zero documentation and the guy that built it left a while ago.
I have never been able to blame our build and code infra for lack of productivity.
I'm seeing issues with repo pages entirely (getting Unicorn "We couldn't respond to your request in time" errors) but the status page hasn't updated to show that yet.
Bitbucket was down for a few hours a week ago and 0 fuss was made about it in the media. Atlassian stock actually went up over the ourage
Besides projects/companies/people stuck on Bitbucket, is anyone actually voluntary using Bitbucket? I remember using it back when I was poor and it was the only choice for hosted free private repositories, but the rest of the platform was actively worse in basically every way compared to the alternatives.
Because literally nobody uses it.
Pun intended?
This continues to be a regular thing for the past 5 years of outages and I've lost count of the many outages that GitHub continues to have since calling it. [0]
[0] https://news.ycombinator.com/item?id=22868406
They're saying PRs and issues but I can't access github.com or github oauth for logging into external services. Need to migrate our org off of using github auth for anything critical I think, this is twice so far this year that we've had these issues.
The industry made a general decision after ZIRP ended to deprioritise availability after years of historic levels of engineering investment.
It's no surprise we're now feeling those effects but damn, GitHub and other services like Slack have been really bad lately.
Or, GitHub used to favor stability back in the day, and there wasn't a lot of changes. People were complaining that GitHub didn't "improve" enough day-to-day, so after the Microsoft purchase, Microsoft started forcing GitHub to add more features, stability be damned.
Most outages are caused not by stuff just randomly breaking, but updates/upgrades going wrong. If you try to increase the output of new features/changes to a platform, you're bound to have more outages and downtime if you aren't more careful than before.
Microsoft, who never really excelled at engineering, to the surprise of absolutely everyone, choose adding features over stability and since years back, we're seeing the consequences of that choice.
I'm not totally certain you can attribute it to this. I know there's a bunch of work going on to migrate things into Azure after the MS acquisition but it feels like more of an industry trend that we cut engineering spend at the cost (often) of lower quality output and outages like these.
GitHub laid off 10% of their staff in 2023 and like you say, won't have slowed down to account for that.
Ignoring all that though... other than Copilot, what big feature changes have you seen in GitHub? My experience of using their product has been broadly unchanged for years.
> My experience of using their product has been broadly unchanged for years.
Compared to how fast/slow they were moving 2012-2018, they're moving at blazing speed now. It feels like every time I open GitHub now, there are new features/changes and "Beta" available stuff. The platform is almost completely different today than it was in 2018, for better or worse.
The move to React was done in 2022. I noticed that one for sure.
GitHub's reliability has gotten pretty terrible since the Microsoft acquisition, well before ZIRP ended.
Citation needed?
Is this true beyond github and slack? Both were acquired btw, that could explain their availability issues (just like twitter, now X).
What about e.g. AWS or GCP? Has their availability meaningfully reduced?
I wouldn't say this applies to cloud providers, they have a very different business on their hands.
But for SaaS in general I think the trend is noticeable? Twitter led the way with massive layoffs in engineering often in the roles around reliability. The industry as a whole have aimed to cut costs however possible, and reliability/ops is usually seen as a cost-centre that gets hit hard.
I've watched this in my own space (start/scale-ups and larger companies, I work in incident response tooling) as people start talking very differently about reliability and engineering investment. You hear "do more with less" about five times every day and spend that was previously greenlit by default around reliability/redundancy is under much more scrutiny now.
I see this as a silent mirror of the reduction in open-source efforts from companies now there's been a refocus on business impact and bottom-line.
If I'm not wrong, this is the third week in a row with GitHub incidents, isn't it?
The incidents history page is really damning. I didn't realize it gotten so bad
Work was impacted by a few outages that they didn’t report there.
Direct incident link: https://www.githubstatus.com/incidents/nm83zrdky73y
The last few days, the “create pr” screen kept hanging for me. I figured out I could reload it, click “create”, grab a coffee, and then come back and it’d be at the next screen.
Maybe it’s time to learn their CLI (and then patiently wait for work to switch to another git service).
Maybe we can spend this downtime researching self-hosting options
I feel like it's not only GitHub... Slack is very slow, too.
I've been getting a couple generic Cloudflare 521 responses across a weird mix of big and not-big sites this morning. Things have felt a little off.
It was me. Sorry. I just merged a PR and down it went :/
Why don't they make the AI fix it? Are they stupid?
It's not just PRs and issues, my build systems are breaking because I can't download tarballs for dependencies from github.
One thing I started doing a few years ago is that, any time I check out a git repo, whether it's on github or anywhere else, is clone it to my NAS and then clone from there to whatever I'm building it on. Aside from guarding against losing dependencies to outages (or from being yanked), it can be handy for guarding against my own fat fingers, and for making sure that the same version is running on multiple systems.
I think that's deepseek striking back.
Everything is slow. It takes several seconds to load any page, submit a comment, etc.
I honestly can't believe how unreliable Github is. Outages are commonplace. It boggles my mind how nothing has been done to address the reliability regressions that have been creeping in ever since MS took over.
Seems like it's back for me (and slow.. and now down again)
First incident of the year for github :tada:
There were others of varying severity: https://www.githubstatus.com/history
At one point all git operations were down: https://news.ycombinator.com/item?id=42691184
Yep, PRs and Issues are affected https://www.githubstatus.com/
That is literally the link submitted here
Actions are clearly affected too but not represented on the status page.
I can't even SSO into one of our orgs because of 500s/timeouts
Yeah SSO is down for me too
even the main github.com page seems down for me now (also getting the unicorn that others saw on their repo)
In a universe, where Obi-Wan Kenobi is a developer: "I felt a great disturbance in the workflow, as if millions of requests suddenly cried out in terror and were suddenly silenced. I fear something terrible has happened."
I just tried to download Peazip, and immediately came here to check if 'this is just me'. Glad I'm not the only one.
meh.
As usual that will be fixed within the 10 to 600 mins.
I blame ai