The goal of this project was to find a way to automate logging into a website (bricklink.com), navigate to a certain page on the website, and save the html source code for that page into a text file. I needed to log into the website because, in the long run, I will be visiting many parts of the website. However, the website will cap out at 1000 requests per session unless I log into a valid account.
To do this, I used Selenium, which is a basic web automation service primarily used for testing the rigidity of the GUIs of a website (read more about it here: https://www.selenium.dev/). I integrated it with a Visual Studio project using the NuGet project manager, so I could easily script the actions using C#.
To begin, I had to find the id element of each part of the login process on the web page. To do this, I had to inspect element on each of them, and find the id parameter in the source code that came up. I then wrote the scrip that inputted my credentials into the web page, logged me in, and redirected me to a desired webpage.
My biggest problem that I ran into was that a cookies prompt was displayed every time I tried to login to the website. It was not as straightforward as finding the id of the accept button and activating it, but I had to dig a little, and find a JavaScript script executor method to run the method embedded in the button that accepts the cookies and hides the prompt.
I finally pulled all of the source code from the page and wrote it to a text file for later. I was first thinking about saving it as a html page so I could read the page I saved using Selenium, similarly to how I got to the page, but there was some problems opening the file on my computer, so I will need to work that out later.