is this question a good fit for this site : python - scraping a site with no feed on daily basis as if it had one

Question

Is the below question a good fit for https://softwareengineering.stackexchange.com/ ? If not, is it worth a place in StackOverflow site ? If not again, in what StackExchange site can I ask it ?

Question Title :Python- scraping news articles on daily basis from sites that do not have any feed

Question body: I can use Python Beautiful Soup module to extract news items from a site feed URL. But suppose the site has no feed and I need to extract news articles from it on daily basis as if it had a feed.

Edit 1: The site https://www.jugantor.com/ has no feed. Even by googling, I did not find any feed of this site . With the following code snippet, I tried to extract the links from the site . The result shows links such as 'http://epaper.jugantor.com' But the news items appearing on the site are nor included in the extracted links.

My Code:

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen
import re


def getLinks(url):

    USER_AGENT = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5'
    request = Request(url)
    request.add_header('User-Agent', USER_AGENT)
    response = urlopen(request)
    content = response.read().decode('utf-8')
    response.close()

    soup = BeautifulSoup(content, "html.parser")
    links = []

    for link in soup.findAll('a', attrs={'href': re.compile("^http://")}):
        links.append(link.get('href'))

    return links

print(getLinks("https://www.jugantor.com/"))

Obviously this does not serve the intended purpose. I need all the news article links of 'https://www.jugantor.com/' on a daily basis as if I acquire them from a feed.
How can I do that ? Any python module or algorithm etc ?

Addendum The question was asked on Stackoverflow in between and got an answer.

Answer 1

I am pretty sure, if you ask this question this way, it would probably be closed as either "too broad" or as being interpreted as a question for programming help, which is off-topic for this site.

In general, asking for an algorithm who does such news scraping could be on topic. However, you should try to

• tell us more details what you already tried, and why it did not suit your needs (your mentioning of "beautiful soup" is a good start for this, but since this module is not dedicated for news feeds, why exactly couldn't you utilize it?)

• tell us which of the solutions you found by googling "python web scraping" you already tried, and why that did not work, either

• try to keep it language agnostic (that does not mean you cannot mention you used Python for your first experiments). Asking for language-specific tools or modules is asking for a 3rd party resource, which is 100% off-topic here.

Askers are expected here to do some research on their own before their post a question. One possible way to do this mght be by actually writing a scraper prototype first, and when you hit some algorithmic road blocks, ask about them. That would probably lead to a much more focussed question.