I'm trying to web-scrape likes on posts on Instagram.
I've tried going through each post but after a certain number of posts Instagram stop responding to the requests.
So now I'm trying to scrape the like count without opening post, i.e when you hover your mouse on a post thumbnail it shows you the like and comment count.
Reference of thumbnail.
For this I use the move_to_element function in this manner:
1 - Search list of posts.
2 - Use move_to_element to hover on those posts.
3 - Scrape Data.
4 - Scroll
Repeat Steps 1-4.
My program goes through the first group without any problem and scrolls. But then starts going through from the 1st post and not from where it stopped or first element loaded after scrolling.
Code (Simplified):
newPost = True
while newPost:
try:
action = webdriver.ActionChains(driver)
newPost = False
var = WebDriverWait ...
container = driver.find_elements .. #GET THE LIST OF POSTS
for post in container:
action.move_to_element(post).perform()
link = post.find_element(By.TAG_NAME, 'a').get_attribute("href")
likes = ...
if link not in postData: #TO CHECK IF ENCOUNTERED NEW POST
postData[link] = likes
newPost = True
driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
sleep(5)