A webmining CLI tool & library for python.

I am trying to run minet in a github action. It fails with the following message:

  minet tw scrape tweets -o tweets.csv "from:@taniki #tutotal2022"
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.9.5/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.5/x64/lib

Collecting tweets: 0 tweets [00:00, ? tweets/s]Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/casanova/reader.py", line 151, in __init__
    fieldnames = next(self.reader)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.5/x64/bin/minet", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/minet/cli/__main__.py", line 218, in main
    fn(cli_args)
  File "/opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/minet/cli/twitter/__init__.py", line 33, in twitter_action
    twitter_scrape_action(cli_args)
  File "/opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/minet/cli/twitter/scrape.py", line 45, in twitter_scrape_action
    enricher = casanova.enricher(
  File "/opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/casanova/enricher.py", line 31, in __init__
    super().__init__(input_file, no_headers=no_headers, **kwargs)
  File "/opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/casanova/reader.py", line 157, in __init__
    raise EmptyFileError
casanova.exceptions.EmptyFileError

Collecting tweets: 0 tweets [00:00, ? tweets/s]
Error: Process completed with exit code 1.

• failed run
• workflow file

Hi, thanks for the last release, I'm glad to see there is a Retweeters tool but I went through some issues with it... for a few days.. I may not understood how it should implemented ? I run it and I get this error : May someone who manage with it help me ?

Thank you

new method to acquire the guest_token through activate API relates #384 #382

Method taken from @JustAnotherArchivist in snscrape see: https://github.com/JustAnotherArchivist/snscrape/commit/0336ce13edbd195b3e91487061a0e7a2857f0c68 Thanks for sharing the solution.

For now this edit is simply a new method to acquire the token. The token is used as a cookie as before but it's not preserved on disk in case of multiple calls.

minet tw scrape tweets '#5gcovid' > tweets.csv

<class 'minet.twitter.exceptions.TwitterPublicAPIInvalidResponseError'>

{'errors': [{'message': 'Over capacity', 'code': 130}]} 503

Hi, when I try to retrieve the retweeters list from a file containing tweets previously extracted from Twitter using minet scrapper, I get this error after scanning a few tweets from my list (after 7, 10, or 30 tweets scanned... it depend of the database...). Does anyone encountered this error before ? Thanks for helping :-)

Lorsque je lance une requête, avec comme critère un mot clé + un utilisateur, le résultat est très aléatoire : une fois 0 tweet, une fois 1 tweet, une fois 20 tweets, une fois 80 tweets etc sans jamais arriver à une extraction totale (qui est d'environ seulement 200 tweets pourtant). J'ai relancé cette requête de nombreuses fois, sans jamais extraire l'ensemble des tweets en question.

Que dois-je faire pour y parvenir ? Merci

Hello,

Thanks for sharing the lib with the community. I am not able to get user tweets , I got the error:

Traceback (most recent call last):
  File "/home/bafou/.local/bin/minet", line 8, in <module>
    sys.exit(main())
  File "/home/bafou/.local/pipx/venvs/minet/lib/python3.8/site-packages/minet/cli/__main__.py", line 198, in main
    to_close = resolve_arg_dependencies(cli_args, config)
  File "/home/bafou/.local/pipx/venvs/minet/lib/python3.8/site-packages/minet/cli/argparse.py", line 290, in resolve_arg_dependencies
    setattr(cli_args, name, value.resolve(config))
  File "/home/bafou/.local/pipx/venvs/minet/lib/python3.8/site-packages/minet/cli/argparse.py", line 253, in resolve
    return getpath(config, self.key, self.default)
  File "/home/bafou/.local/pipx/venvs/minet/lib/python3.8/site-packages/ebbe/utils.py", line 72, in getpath
    target = target[step]
TypeError: string indices must be integers

when executingminet tw user-tweets screen_name users.csv > tweets.csv with users.csv

Regards.

hi,

i have this GH action to generate a twitter scrap csv (written by @taniki) :

name: scrape bfm

on:
  workflow_dispatch:
  schedule:
    - cron:  '0 9 * * *'

jobs:
  scrape_bfm:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/[email protected]
      - uses: actions/[email protected]
        with:
          python-version: '3.x'
      - name: install minet
        run: |
          python -m pip install --upgrade pip
          pip install minet==0.56.2
      - name: scrape @BFMTV tweets
        shell: bash
        run: |
          minet tw scrape tweets "from:@BFMTV since:2021-09-01" > bfmtv-tweets.csv
      - name: commit
        uses: ./.github/actions/commit
        with:
          message: lol @bfmtv

Sometimes, no problem. Sometimes, GH return error log :

Run minet tw scrape tweets "from:@CNEWS since:2021-09-01" > cnews-tweets.csv
Collecting tweets: 0 tweets [00:00, ? tweets/s]                            
Collecting tweets: 0 tweets [00:00, ? tweets/s]                   
Searching for "from:@CNEWS since:2021-09-01"

Collecting tweets: 0 tweets [00:00, ? tweets/s]
Collecting tweets: 0 tweets [00:00, ? tweets/s, queries=1, tokens=1]Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.1/x64/bin/minet", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/site-packages/minet/cli/__main__.py", line 218, in main
    fn(cli_args)
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/site-packages/minet/cli/twitter/__init__.py", line 31, in twitter_action
    twitter_scrape_action(cli_args)
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/site-packages/minet/cli/twitter/scrape.py", line 69, in twitter_scrape_action
    for tweet, meta in iterator:
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/site-packages/minet/twitter/api_scraper.py", line 370, in search
    new_cursor, tweets = retryer(self.request_search, query, cursor, refs=refs)
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/site-packages/tenacity/__init__.py", line 404, in __call__
    do = self.iter(retry_state=retry_state)
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/site-packages/tenacity/__init__.py", line 349, in iter
    return fut.result()
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/site-packages/tenacity/__init__.py", line 407, in __call__
    result = fn(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/site-packages/minet/twitter/api_scraper.py", line 72, in wrapped
    self.acquire_guest_token()
  File "/opt/hostedtoolcache/Python/3.10.1/x64/lib/python3.10/site-packages/minet/twitter/api_scraper.py", line 261, in acquire_guest_token
    raise TwitterGuestTokenError
minet.twitter.exceptions.TwitterGuestTokenError

Collecting tweets: 0 tweets [00:00, ? tweets/s, queries=1, tokens=1]
Error: Process completed with exit code 1.

Dont understand. Did anyone have the same problem Twitter ban GH sometimes ?

Thanks for Minet, super outil !

Forewords : sorry, new on GitHub, and I'm not sure it is the appropriate place to post my question... Is it ?

Hi, First, thank you for the tool which will help me a lot in my research ! I got a problem, which I think is not that complicated, but when I run Minet in order to get the "friends" of the twitter_users contained in the data_users.csv file, I don't manage to get access to the file : "Permission Denied"... I tried to open the CMD as an Administrator but it didn't solve the problem. Can you help me ?

while installing mineit via pip it does not work. says, "" Collecting mineit Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://pypi.org/simple/mineit/

""

is this issue already solved?

As in #382 I experience systematic TwitterGuestTokenError exceptions. Was not the case a few weeks ago. I didn't test other versions than 0.56.1 and 0.56.2.

Looks like we need to review the twitter scrape heuristic. I will try to have a look later today or tomorrow.

• [ ] get comments from a post id: https://www.instagram.com/api/v1/media/POST_ID/comments/?can_support_threading=true&permalink_enabled=false
• [x] get user info from username: https://i.instagram.com/api/v1/users/web_profile_info/?username=USERNAME
• [ ] other route for posts associated with hashtag (more info but don't know how to change page): https://www.instagram.com/api/v1/tags/web_info/?tag_name=HASHTAG
• [ ] get post info from post id: https://www.instagram.com/api/v1/media/POST_ID/info/
• [ ] get post likers from post id (it seems that we can only have access to a limited number of them): https://www.instagram.com/api/v1/media/POST_ID/likers/

Need 'cookie' and 'x-ig-app-id'