japhyr 22 days ago

There's a not-entirely-small part of me that hopes Issues breaks entirely for long enough that people at GitHub rethink their implementation.

GH issues have been so useful for the better part of a decade, that I have an empty repo called my_life just so I can make issues about things like home maintenance. In the last few months, the UI has become so slow and flaky that I don't use it for nearly as much as I used to. And when I have to for a project, it slows me down noticeably.

I thought I was alone in this, but asking around I found that this is a common frustration among developers I communicate regularly with.

  • simonw 22 days ago

    I do the same - I've been running so many non-software things out of private GitHub Issues. I even have a simple system for creating a new issue every day to use as a personal TODO list, described here: https://til.simonwillison.net/github-actions/daily-planner

    I've noticed everything feeling a whole lot less snappy and responsive over the past few days - ironically I think it's because they've done a major rewrite of the frontend presumably with the aim of making it more snappy and responsive!

  • PapaPalpatine 22 days ago

    I’m low-key certain it’s because they started using more React on the front end, especially for the new Issue stuff. The entire UI feels slow and janky like GitLab.

  • straws 22 days ago

    You're not alone. There's been an effort to transparently update the UI to a React implementation over the past year or two, and while I understand the benefits to that approach, they have introduced some flakiness in moving away from a the server-rendered pjax/html-pipeline/simple web components approach that was so cohesive and battle tested over the decade before it.

  • jrochkind1 22 days ago

    I am not experiencing a full outage right now, but I am experiencing very slow (spinner) and flakey (error that works again on repeat).

    So perhaps it's the same problem, that's been getting more prevalent until it was noticeable by monitoring. One can hope!

  • RohMin 22 days ago

    That's actually a really interesting way to leverage that feature. Have you found this easier than other services built specifically for this use case?

    • japhyr 22 days ago

      I've tried other tools. But I'm on GitHub most days, so it's been a seamless way to keep track of some things that would otherwise disappear into a calendar, or some tool that I don't use as often.

  • joseda-hg 22 days ago

    For something so critical, does it make sense to self host say a gitea instance or such?

    I know you can sync repos, I don't think you can sync issues tho

    • snapplebobapple 22 days ago

      this is what i doits worth the 11 dollars a year for the vps to control my own

bigfatfrock 22 days ago

Who's going to kick off the holy war around self hosting?

  • reddalo 22 days ago

    Speaking of self hosting, Codeberg [1] is great. It's open and it looks and behaves pretty much like GitHub (unlike GitLab).

    [1] https://codeberg.org/

    • diggan 22 days ago

      Seconded. Hosting FOSS on a platform that isn't FOSS itself and run by a for-profit company who tried to work against FOSS for so long (Microsoft) doesn't make much sense to me.

      Instead, we should dogfood the FOSS ecosystem on a platform that is FOSS itself, and run as a non-profit. Codeberg, for better or worse, is the best platform for this today.

    • mr_mitm 22 days ago

      Did you mean to say Forgejo? Correct me if I'm wrong, but Codeberg is simply hosting a Forgejo instance for you, so not exactly self hosting.

  • skydhash 22 days ago

    Personal projects are on a vps and accessed through ssh. I’ve self-hosted gitea, then forgejo, then found out that I don’t like the interface or the auth dance. My plan is to ise cgit if I want something to be public and any forge if the purpose is collaboration (sourcehut is nice)

  • duxup 22 days ago

    Honestly, sounds like a lot of work and I'm not sure I get much benefit, or even less downtime.

    I don't know about others but for me github is hardly a service that I'm sweating moment to moment uptime. My apps are still running and so on.

  • cropcirclbureau 22 days ago

    Hosting GitLab is not fun but anyways, M$ can still suck it.

    • zaruvi 22 days ago

      Not Onedev or Forgejo, though it could get tricky depending on what features you require.

  • rvz 22 days ago

    It all began 5 years ago [0] when we wanted to 'centralize everything to GitHub.'

    Then, the outages accelerated from there.

    [0] https://news.ycombinator.com/item?id=22868406

    • mrguyorama 22 days ago

      "are you sure your self-hosted solution will have better uptime?"

      Hilariously, after 15 years working in self hosted bitbucket systems, YES entirely.

      An underfunded university with an incompetent but doing their best IT department? Zero downtime.

      A mid sized company full of overly confident and "just build it" nerds building fragile shit? Zero downtime.

      A large corporation with a completely outsourced IT department that can't give you access to something unless you do exactly the right undocumented thing in our internal ticketing software? Zero downtime.

      That includes self hosted jenkins and literally homebuilt infrastructure with zero documentation and the guy that built it left a while ago.

      I have never been able to blame our build and code infra for lack of productivity.

peterldowns 22 days ago

I'm seeing issues with repo pages entirely (getting Unicorn "We couldn't respond to your request in time" errors) but the status page hasn't updated to show that yet.

cjonas 22 days ago

Bitbucket was down for a few hours a week ago and 0 fuss was made about it in the media. Atlassian stock actually went up over the ourage

  • diggan 22 days ago

    Besides projects/companies/people stuck on Bitbucket, is anyone actually voluntary using Bitbucket? I remember using it back when I was poor and it was the only choice for hosted free private repositories, but the rest of the platform was actively worse in basically every way compared to the alternatives.

  • jeffbee 22 days ago

    Because literally nobody uses it.

  • crest 22 days ago

    Pun intended?

G1N 22 days ago

They're saying PRs and issues but I can't access github.com or github oauth for logging into external services. Need to migrate our org off of using github auth for anything critical I think, this is twice so far this year that we've had these issues.

lawrjone 22 days ago

The industry made a general decision after ZIRP ended to deprioritise availability after years of historic levels of engineering investment.

It's no surprise we're now feeling those effects but damn, GitHub and other services like Slack have been really bad lately.

  • diggan 22 days ago

    Or, GitHub used to favor stability back in the day, and there wasn't a lot of changes. People were complaining that GitHub didn't "improve" enough day-to-day, so after the Microsoft purchase, Microsoft started forcing GitHub to add more features, stability be damned.

    Most outages are caused not by stuff just randomly breaking, but updates/upgrades going wrong. If you try to increase the output of new features/changes to a platform, you're bound to have more outages and downtime if you aren't more careful than before.

    Microsoft, who never really excelled at engineering, to the surprise of absolutely everyone, choose adding features over stability and since years back, we're seeing the consequences of that choice.

    • lawrjone 22 days ago

      I'm not totally certain you can attribute it to this. I know there's a bunch of work going on to migrate things into Azure after the MS acquisition but it feels like more of an industry trend that we cut engineering spend at the cost (often) of lower quality output and outages like these.

      GitHub laid off 10% of their staff in 2023 and like you say, won't have slowed down to account for that.

      Ignoring all that though... other than Copilot, what big feature changes have you seen in GitHub? My experience of using their product has been broadly unchanged for years.

      • diggan 22 days ago

        > My experience of using their product has been broadly unchanged for years.

        Compared to how fast/slow they were moving 2012-2018, they're moving at blazing speed now. It feels like every time I open GitHub now, there are new features/changes and "Beta" available stuff. The platform is almost completely different today than it was in 2018, for better or worse.

      • mariusor 22 days ago

        The move to React was done in 2022. I noticed that one for sure.

  • rohansingh 22 days ago

    GitHub's reliability has gotten pretty terrible since the Microsoft acquisition, well before ZIRP ended.

  • NewJazz 22 days ago

    Citation needed?

    Is this true beyond github and slack? Both were acquired btw, that could explain their availability issues (just like twitter, now X).

    What about e.g. AWS or GCP? Has their availability meaningfully reduced?

    • lawrjone 22 days ago

      I wouldn't say this applies to cloud providers, they have a very different business on their hands.

      But for SaaS in general I think the trend is noticeable? Twitter led the way with massive layoffs in engineering often in the roles around reliability. The industry as a whole have aimed to cut costs however possible, and reliability/ops is usually seen as a cost-centre that gets hit hard.

      I've watched this in my own space (start/scale-ups and larger companies, I work in incident response tooling) as people start talking very differently about reliability and engineering investment. You hear "do more with less" about five times every day and spend that was previously greenlit by default around reliability/redundancy is under much more scrutiny now.

      I see this as a silent mirror of the reduction in open-source efforts from companies now there's been a refocus on business impact and bottom-line.

brunocroh 22 days ago

If I'm not wrong, this is the third week in a row with GitHub incidents, isn't it?

  • floydian10 22 days ago

    The incidents history page is really damning. I didn't realize it gotten so bad

    • hedora 22 days ago

      Work was impacted by a few outages that they didn’t report there.

hedora 22 days ago

The last few days, the “create pr” screen kept hanging for me. I figured out I could reload it, click “create”, grab a coffee, and then come back and it’d be at the next screen.

Maybe it’s time to learn their CLI (and then patiently wait for work to switch to another git service).

ferriswil 22 days ago

Maybe we can spend this downtime researching self-hosting options

shafyy 22 days ago

I feel like it's not only GitHub... Slack is very slow, too.

  • smitelli 22 days ago

    I've been getting a couple generic Cloudflare 521 responses across a weird mix of big and not-big sites this morning. Things have felt a little off.

xenospn 22 days ago

It was me. Sorry. I just merged a PR and down it went :/

adhamsalama 22 days ago

Why don't they make the AI fix it? Are they stupid?

tovej 22 days ago

It's not just PRs and issues, my build systems are breaking because I can't download tarballs for dependencies from github.

  • flyinghamster 22 days ago

    One thing I started doing a few years ago is that, any time I check out a git repo, whether it's on github or anywhere else, is clone it to my NAS and then clone from there to whatever I'm building it on. Aside from guarding against losing dependencies to outages (or from being yanked), it can be handy for guarding against my own fat fingers, and for making sure that the same version is running on multiple systems.

dexcs 22 days ago

I think that's deepseek striking back.

khaledh 22 days ago

Everything is slow. It takes several seconds to load any page, submit a comment, etc.

rnicholus 22 days ago

I honestly can't believe how unreliable Github is. Outages are commonplace. It boggles my mind how nothing has been done to address the reliability regressions that have been creeping in ever since MS took over.

kalleboo 22 days ago

Seems like it's back for me (and slow.. and now down again)

cristaloleg 22 days ago

Yep, PRs and Issues are affected https://www.githubstatus.com/

  • malfist 22 days ago

    That is literally the link submitted here

  • sodapopcan 22 days ago

    Actions are clearly affected too but not represented on the status page.

  • pera 22 days ago

    I can't even SSO into one of our orgs because of 500s/timeouts

    • fron 22 days ago

      Yeah SSO is down for me too

rcleveng 22 days ago

even the main github.com page seems down for me now (also getting the unicorn that others saw on their repo)

ahofmann 22 days ago

In a universe, where Obi-Wan Kenobi is a developer: "I felt a great disturbance in the workflow, as if millions of requests suddenly cried out in terror and were suddenly silenced. I fear something terrible has happened."

theandrewbailey 22 days ago

I just tried to download Peazip, and immediately came here to check if 'this is just me'. Glad I'm not the only one.

sixthDot 22 days ago

meh.

As usual that will be fixed within the 10 to 600 mins.