FlexGet

From Indie IT Wiki

FlexGet is a multi-purpose automation tool for content like torrents, nzbs, podcasts, comics, series, movies, etc. It is able to handle different kinds of sources like RSS-feeds, html pages and csv files. There are even some plugins for sites that do not provide any kind of useful feeds.

There are numerous plugins that allow utilizing FlexGet in interesting ways and more are being added continuously.

FlexGet is extremely useful in conjunction with applications which have watch directory support or provide interface for external utilities like FlexGet.

HOWTO:

Install

NEWER

UPDATE - JANUARY 2021

There is a new Transmission Python module called 'transmission-rpc' which has superceded 'transmissionrpc' and now needs to be installed. 

~/flexget3/bin/pip3.6 install --upgrade pip flexget transmissionrpc transmission-rpc

If you want to have the latest Flexget version 3 then you need to have Python 3.6 installed.

https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa?field.series_filter=xenial

These commands show how to create a new Python virtual environment alongside your current one, and then copy back your database. 

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install python3.6-venv
apt list --installed python*
which python python3 python3.6

python3.6 --version
python3.6 -m venv ~/flexget3/
cd ~/flexget3/
bin/pip3.6 install --upgrade pip
bin/pip3.6 install --upgrade flexget
~/flexget3/bin/flexget check

NEW

Python 2 will be deprecated in January 2020, so you need to switch to using Python 3.

Backup your config just in case... 

cp -av ~/.flexget/ ~/.flexget_BACKUP

Install Python 3... 

sudo apt-get install python3

Check Python 3 version... 

python3 --version

Install the Python3 Virtual Environment... 

sudo apt-get install python3-venv

Create the virtual environment for FlexGet... 

python3 -m venv ~/flexget/

Upgrade Python's PIP software... 

~/flexget/bin/pip install --upgrade pip

Install FlexGet... 

~/flexget/bin/pip install flexget

Check FlexGet version... 

~/flexget/bin/flexget --version

Start FlexGet as a daemon... 

~/flexget/bin/flexget daemon start --autoreload-config --daemonize

Check FlexGet configuration... 

~/flexget/bin/flexget check

Update your crontab files... 

@daily ~/flexget/bin/pip install --upgrade flexget
@reboot ~/flexget/bin/flexget daemon start --autoreload-config --daemonize

https://flexget.com/InstallWizard/Linux

OLD 

sudo -i
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py
pip install certifi
pip install urllib3
pip install flexget

OLDER 

sudo apt-get install python-yaml python-feedparser python-setuptools
sudo easy_install pip
sudo pip install flexget

HOWTO: CHECK:

Installed Version

flexget --version

Backlog

flexget backlog list

Recent Download History

flexget history --limit 10

10 is an arbitrary figure...

Configuration

flexget check

Debug:

flexget --loglevel verbose execute --task task-name --dump

Required Upgrading Actions:

If after upgrading things stop working check the following page for changes made to the config file formatting: 

http://flexget.com/UpgradeActions

HOWTO: EXECUTE

Test

Executes all feeds, but doesn't write anything into disk. 

flexget --test execute

Actual

flexget execute now

Automate

OLD method... use new DAEMON method below this. 

crontab -e
@hourly /usr/local/bin/flexget --cron

In Daemon Mode On System Startup

@reboot /usr/local/bin/flexget daemon start -d

HOWTO: UPDATE:

Update Flexget

This is the new method, which fixes dependencies and starts afresh :-)

If you receive any errors, it's probably because you have LOTS of old FlexGet install files!

See also about updating PYTHON - Python

NEW 

sudo wget https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
which pip    (/usr/local/bin/pip)
sudo pip list
sudo pip install --upgrade setuptools
sudo pip install --upgrade pip
sudo rm -rfv /usr/local/lib/python2.7/dist-packages/flexget*
sudo rm -rfv /usr/local/lib/python2.7/dist-packages/FlexGet*
sudo pip install --upgrade transmissionrpc
sudo pip install --upgrade flexget

OLD 

sudo easy_install --upgrade flexget

HOWTO: SERIES:

List Simple

flexget series list --table-type porcelain | awk -F '|' '{ print $1 }'

List All Series In Queue

flexget series list

Forget An Entire Show

flexget series forget "Show Name"

Start A Specified Episode Of A Show

flexget series begin "Show Name" S02E04

Series Help

~/flexget/bin/flexget series --help

usage: flexget series [-h] <action> ...

optional arguments:
 -h, --help  show this help message and exit

actions:
 <action>
   list      List a summary of the different series being tracked
   show      Show the releases FlexGet has seen for a given series
   begin     set the episode to start getting a series from
   add       Add episode(s) and season(s) to series history
   forget    Removes episodes or whole series from the entire database
   remove    Removes episodes or whole series from the series database only

Series Add An Episode

If FlexGet is not adding episodes, it's probably got too far behind behind, so you can manually add episode to 'catch up'... 

~/flexget/bin/flexget series add 'Castle Rock' 'S02E05' --quality 720p

Force A Learn

flexget inject "Show Name S02E04" --force --accept --learn

Show Progress Of A Series

flexget series show "Gotham"

Search In Seen Database For Show

flexget seen search "Gotham"

Forget Torrent URL Used For Episode

flexget seen forget "http://itorrents.org/torrent/9F499.torrent?title=Gotham-S03E01-720p-HDTV-350MB--NBY"

Inject Torrent URL Used For Episode

flexget inject "http://itorrents.org/torrent/9F499.torrent?title=Gotham-S03E01-720p-HDTV" --force --accept

Inject Torrent URL Used For Episode DUMP Output

~/flexget3/bin/flexget --test inject "https://zoink.ch/torrent/Dark.S03E01.720p.HEVC.x265-MeGusta[eztv.io].mkv.torrent" --tasks eztvio-tv --dump

-- Accepted: ---------------------------
title            : Dark.S03E01.720p.HEVC.x265-MeGusta[eztv.io].mkv.torrent
url              : https://zoink.ch/torrent/Dark.S03E01.720p.HEVC.x265-MeGusta[eztv.io].mkv.torrent
original_url     : https://zoink.ch/torrent/Dark.S03E01.720p.HEVC.x265-MeGusta[eztv.io].mkv.torrent
media_id         : <LazyField - value will be determined when it is accessed>
original_title   : Dark.S03E01.720p.HEVC.x265-MeGusta[eztv.io].mkv.torrent
proper           : False
proper_count     : 0
quality          : 720p h265
release_group    : None
season_pack      : None
series_episode   : 1
series_episodes  : 1
series_exact     : True
series_id        : S03E01
series_id_type   : ep
series_identified_by: ep
series_name      : Dark (2017)
series_parser    : <SeriesParseResult(data=Dark.S03E01.720p.HEVC.x265-MeGusta[eztv.io].mkv.torrent,name=Dark (2017),id=(3, 1),season=3,season_pack=None,episode=1,quality=720p h265,proper=0,special=False,status=OK)>
series_releases  : [29494]
series_season    : 3
task             : eztvio-tv
tvdb_id          : 334824

Use Directory Contents To Populate Series Database

 populate-series-db:
   priority: 2
   template:
     - series-metainfo
   seen: local
   filesystem:
     # path: "/{? folder.root ?}{? folder.series ?}"
     path: "/home/user/Plex/TV/"
     recursive: yes
     retrieve: files
     regexp: '.*\.(avi|mkv|mp4)$'
   accept_all: yes
   configure_series:
     from:
       filesystem:
         # path: "/{? folder.root ?}{? folder.series ?}"
         path: "/home/user/Plex/TV/"
         retrieve: dirs
     settings:
       tracking: no
       identified_by: ep

https://discuss.flexget.com/t/mark-series-as-seen-based-on-folder-contents/ 

flexget --test execute --task populate-series-db

INFO: INPUT METHOD:

Directory Folders As Input List

Check exact name...

http://thetvdb.com/?tab=series&id=272644&lid=7

Create folder... 

mkdir /home/user/Videos/TV/"12 Monkeys"

List folder... 

tree -d /home/user/Videos/TV/

/home/user/Videos/TV/
├── 12 Monkeys
├── Almost Human
├── American Horror Story
├── Archer (2009)
├── Backstrom
├── Bad Education

Configure FlexGet... 

   configure_series:
     settings:
       target: 720p
       timeframe: 6 hours
       quality: hdtv+ <=720p
     from:
       listdir: /home/user/Videos/TV/

Trakt As An Input List

First, you need to authenticate with Trakt manually on the command line. Running this command will give you a URL to go to and a code to type in on that page to authorize your account. It will wait for you to do so. When this is done, an access_token is generated and stored... 

flexget trakt auth <account>

To check your current authentication token... 

flexget trakt list <account>
There is a FlexGet process already running for this config, sending execution there.
Authorization expires on 2021-02-12 12:25:38.711268

https://flexget.com/Trakt_Authentication

Here is a code block example, using 2 different lists... 

   configure_series:
     settings:
       target: 720p
       timeframe: 6 hours
       quality: hdtv+ <=720p
     from:
       inputs:
       - trakt_list:
           username: user
           strip_dates: yes
           list: watched
           type: shows
       - trakt_list:
           username: user
           strip_dates: yes
           list: watchlist
           type: shows

INFO:

Changelog

https://flexget.com/ChangeLog

GitHub

https://github.com/Flexget/Flexget

CookBook

Recipes

Configuration

In order to use FlexGet you'll need to create a configuration file.

FlexGet uses YAML markup in configuration file. Here are some common mistakes:-

  • Indentation level. Always use (multiples of) 2 spaces and never use tab-key!
  • Plugins are supposed to be indented at the same level (rss, series, download etc). Some plugins may allow other plugins inside of them.
  • Missing colons. Pay special attention to these when looking at the examples and documentation.
  • If text value contains any of {}[]%: characters it must be quoted with .
  • If you want to pass a number as a text (ie. series 24), value must be quoted with .

Here is our example configuration: 

~/.flexget/config.yml

feeds:
  tv-shows:
    rss: http://example.com/rss.xml
    series:
      - pioneer one
      - south park
    download: ~/torrents/

Now let's break that down line by line.

In first line feeds is a container which may contain any number of feeds under it. In our example there is only one feed called tv-shows. The feed name must be indented by 2 spaces because it belongs to feeds. In Yaml relations are represented by indentation level. Pay special attention to : - characters.

Fully Working Configuration Example

presets:
  tv:
    series:
      settings:
        720p:
          quality: 720p
        hdtv:
          quality: hdtv
      720p:
        - The Looney Tunes Show
        - Awake
      hdtv:
        # - Camelot:
            # watched: S01E03
        - The Devil You Know
        - Leverage
        - Wilfred
    email:
      from: name@myemail.com
      to:
        - me@myemailaddress.com
        - me@anotheremailaddress.com
    download: ~/Downloads/
feeds:
  ezrss-all:
    rss: http://www.ezrss.it/feed/
    preset: tv
  showrss-all:
    rss: http://showrss.karmorra.info/feeds/all.rss
    preset: tv
  # ezrss-camelot:
    # rss: http://www.ezrss.it/search/index.php?show_name=Camelot&mode=rss
    # preset: tv

Documentation

flexget --help
flexget --doc series

Thanks to the Flexget wiki

INFO: PLUGINS:

https://flexget.com/Plugins

Our feed tv-shows has three plugins: rss, series, download. Notice all plugins again indented by additional 2 spaces. This is because they belong to the feed tv-shows. In correct configuration file all plugin keywords appear in the same level.

Common misconceptions

  • Plugin order doesn't matter, you can list them in any order you like. Most logical order would be inputs -> filters -> outputs.
  • Feed order doesn't matter, feeds are executed in seemingly random order. Use priority plugin prioritize feeds when necessary.
  • You cannot list plugins, i.e. rss twice in a single feed. This is limited by the chosen simpler syntax in the configuration file. Instead see preset plugin.

Sceper / Scene Releases

RSS

This is our input plugin which reads a RSS feed and produces processable entries. By looking at plugin rss documentation we learn that it accepts URL as a parameter in its simplest form.

Series

This plugin expects a list of series names as a parameter. Notice again how the list is intended by 2 spaces more than series keyword. The list belongs to series plugin. This filters entries, it goes through all of them and accepts those which match our interest.

Download

The process ends at output, in this case we have single output that just downloads all accepted entries into given path.

UPDATE - does not appear to work. Change to EZTV (see below).

Syntax:- 

scenereleases: http://sceper.eu/category/tv-shows/tv-shows-x264?themedemo=SceneRLSv3

Full working configuration file example:- 

feeds:
  tv-shows:
    scenereleases: http://sceper.eu/category/tv-shows/tv-shows-x264?themedemo=SceneRLSv3
    series:
      - The.Looney.Tunes.Show.2011.*720p
    download: ~/Downloads/

RSS + EZTV

feeds:
  tv-shows:
    # this is the actual feed you want, uncomment when ready
    # rss: http://www.ezrss.it/feed/
    # this is a test feed to see if it works, delete this when ready
    rss: http://www.ezrss.it/search/index.php?show_name=Camelot&mode=rss
    series:
      - Camelot
          # only anything newer than this will be downloaded
          watched: S01E03
    quality:
      min: hdtv
      max: 720p
    download: ~/Downloads/

Transmission RPC

Instead of downloading the .torrent file to a directory for Transmission to 'watch', this plugin passes the .torrent file to the Transmission daemon directly via RPC. Also, it supports the newer magnet urls.

First, install the Python Transmission RPC software: 

sudo easy_install transmissionrpc

or 

sudo pip install transmissionrpc

Then, update your FlexGet configuration file to include the new settings, such as transmission and set: path (which needs to be a directory writeable by transmission-daemon) as shown in the example below. Notice that you need to comment out the download option now which has been replaced by the set: path option. 

presets:
  global:
    # download: ~/Downloads/
    transmission:
      host: localhost
      port: 9091
      username: username
      password: password
  tv:
    series:
      settings:
        shows:
          timeframe: 6 hours
          quality: 720p
          set:
            path: ~/Videos/transmission/seeding/

Now if you run a check you should see something like the following. 

flexget --check

2012-02-17 23:40 INFO     transmission  feed-name      Trying to connect to transmission...
2012-02-17 23:40 INFO     transmission  feed-name      Successfully connected to transmission.

HOWTO: FIX:

PIP Problems

https://flexget.com/PipProblems

ERROR: pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection InsecurePlatformWarning

sudo -i
pip install urllib3[secure]
pip install --upgrade flexget

https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

https://urllib3.readthedocs.io/en/latest/user-guide.html#ssl-py2

ERROR: Series Behind

Show... 

flexget series list

Error... 

"! Latest download is 3 episodes behind, this may require manual intervention"

Fix... 

flexget series begin "Kung Fu Panda: Legends of Awesomeness" S03E17

ERROR: pkg_resources.VersionConflict: (six 1.5.2

pkg_resources.VersionConflict: (six 1.5.2 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('six>=1.7.0'))

Fix... 

sudo easy_install six==1.8.0

ERROR: BUG: Unhandled error in plugin transmission

2014-02-13 19:26 ERROR    task  kat-tv  BUG: Unhandled error in plugin transmission: 'Client' object has no attribute 'add_torrent'
Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/FlexGet-1.2-py2.6.egg/flexget/task.py", line 409, in __run_plugin
   return method(*args, **kwargs)

Fix... 

easy_install --upgrade Flexget
easy_install --upgrade transmissionrpc

ERROR: No module named pkg_resources

sudo -i
curl -O http://python-distribute.org/distribute_setup.py
python distribute_setup.py
easy_install itsdangerous
easy_install markupsafe

ERROR: raise DistributionNotFound(req)

easy_install requests==0.10.0
sudo easy_install SQLAlchemy==0.7.4

http://flexget.com/ticket/1488

ERROR: pkg_resources.DistributionNotFound: python-dateutil<2.0

sudo easy_install python-dateutil==1.5

ERROR: pkg_resources.DistributionNotFound: FeedParser>=5.1.2

sudo easy_install FeedParser==5.1.2

ERROR: pkg_resources.DistributionNotFound: argparse

sudo easy_install argparse

Rejection

If FlexGet rejects your rules like this 

tv-shows     Accepted: 0 (Rejected: 15 Undecided: 0 Failed: 0)

and you want to tell it to try them again, then run... 

flexget --series-forget "Camelot"

Removed series `Camelot` from database.

Extracting Multi-Volume RAR Files

http://wiki.indie-it.com/index.php?title=Rar 

RARFILE=$( ls *.rar ); echo "$RARFILE"; MKVFILE=$( rar l "$RARFILE" |grep 'mkv' |gawk '{ print $1 }' ); echo "$MKVFILE"; rar e "$RARFILE"; sleep 1s; mv "$MKVFILE" ~/Videos/

HOWTO: List Series In Config File

nano ~/.flexget/config.yml

    series:
   # START
     - APB  
     - Archer (2009)
     - Arrow  
     - Berlin Station
     - Big Little Lies
   # END

cat ~/.flexget/config.yml | sed -n '/\# START/,/\# END/p' | grep '-' | sed 's/^ *- //g'
Retrieved from ‘https://wiki.indie-it.com/index.php?title=FlexGet&oldid=197’