Take data from any website 😲 I wanted to take the entire YCombinator startup directory and analyze it using various tools including LLMs and I couldnt find a way to download or export it until I remembered... Im a Web Developer! I know a few tricks up my sleeve. It was actually super easy because the browser calls for all the data in a single request. No authentication or pagination required. EASY #software #hacks #tipsandtricks #webdevelopment #technology #howto

Let me show you how to take data from a website and use it for your own purposes. So I'm here on Y Combinator's website, but this works on a lot of other sites as well. Now what I really want is to download this entire startup directory, but instead of copying and pasting all the values one by one, chances are your browser is already doing a lot of the work. Right click and inspect, go over to the network tab and refresh. If you click this button right here, then you'll see all the calls for additional metadata. For example, this call is all the different metadata for the Y Combinator batches, regions, and industries. And this call right here is all the metadata for all the companies in the startup directory. I'm just going to right click, copy, and copy response. And I have 85,000 lines of metadata. For example, here's all the metadata for Airbnb. It's location, it's industry, launch time, tags. So try that out next time you want to take a lot of data from websites and let me know which other sites it works well on. Follow for more, I'm posting every day. Peace.