FlexGet

From Indie IT Wiki
Jump to: navigation, search

FlexGet is a multi-purpose automation tool for content like torrents, nzbs, podcasts, comics, series, movies, etc. It is able to handle different kinds of sources like RSS-feeds, html pages and csv files. There are even some plugins for sites that do not provide any kind of useful feeds.

There are numerous plugins that allow utilizing FlexGet in interesting ways and more are being added continuously.

FlexGet is extremely useful in conjunction with applications which have watch directory support or provide interface for external utilities like FlexGet.

HOWTO:

Install

NEW

sudo -i
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py
pip install certifi
pip install urllib3
pip install flexget

OLD

sudo apt-get install python-yaml python-feedparser python-setuptools
sudo easy_install pip
sudo pip install flexget

HOWTO: CHECK:

Installed Version

flexget --version

Backlog

flexget backlog list

Recent Download History

flexget history --limit 10

10 is an arbitrary figure...

Configuration

flexget check

Debug:

flexget --loglevel verbose execute --task task-name --dump

Required Upgrading Actions:

If after upgrading things stop working check the following page for changes made to the config file formatting:

http://flexget.com/UpgradeActions

HOWTO: EXECUTE

Test

Executes all feeds, but doesn't write anything into disk.

flexget --test execute

Actual

flexget execute now

Automate

crontab -e

@hourly /usr/local/bin/flexget --cron

In Daemon Mode On System Startup

@reboot /usr/local/bin/flexget daemon start -d

HOWTO: UPDATE:

Update Flexget

This is the new method, which fixes dependencies and starts afresh :-)

If you receive any errors, it's probably because you have LOTS of old FlexGet install files!

ERROR    plugin    cannot import name register_plugin

http://flexget.com/ticket/3042#comment:9

See also about updating PYTHON - Python

NEW

sudo wget https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
which pip    (/usr/local/bin/pip)
sudo pip list
sudo pip install --upgrade setuptools
sudo pip install --upgrade pip
sudo rm -rfv /usr/local/lib/python2.7/dist-packages/flexget*
sudo rm -rfv /usr/local/lib/python2.7/dist-packages/FlexGet*
sudo pip install --upgrade transmissionrpc
sudo pip install --upgrade flexget

OLD

sudo easy_install --upgrade flexget

HOWTO: SERIES:

List Simple

flexget series list --table-type porcelain | awk -F '|' '{ print $1 }'

List All Series In Queue

flexget series list

Forget An Entire Show

flexget series forget "Show Name"

Start A Specified Episode Of A Show

flexget series begin "Show Name" S02E04

Force A Learn

flexget inject "Show Name S02E04" --force --accept --learn

Show Progress Of A Series

flexget series show "Gotham"

Search In Seen Database For Show

flexget seen search "Gotham"

Forget Torrent URL Used For Episode

flexget seen forget "http://itorrents.org/torrent/9F499.torrent?title=Gotham-S03E01-720p-HDTV-350MB--NBY"

Inject Torrent URL Used For Episode

flexget inject "http://itorrents.org/torrent/9F499.torrent?title=Gotham-S03E01-720p-HDTV" --force --accept

INFO: INPUT METHOD:

Directory Folders As Input List

Check exact name...

http://thetvdb.com/?tab=series&id=272644&lid=7

Create folder...

mkdir /home/user/Videos/TV/"12 Monkeys"

List folder...

tree -d /home/user/Videos/TV/

/home/user/Videos/TV/
├── 12 Monkeys
├── Almost Human
├── American Horror Story
├── Archer (2009)
├── Backstrom
├── Bad Education

Configure FlexGet...

   configure_series:
     settings:
       target: 720p
       timeframe: 6 hours
       quality: hdtv+ <=720p
     from:
       listdir: /home/user/Videos/TV/

Trakt As An Input List

   configure_series:
     settings:
       target: 720p
       timeframe: 6 hours
       quality: hdtv+ <=720p
     from:
       inputs:
       - trakt_list:
           username: user
           strip_dates: yes
           list: watched
           type: shows
       - trakt_list:
           username: user
           strip_dates: yes
           list: watchlist
           type: shows

INFO:

Changelog

http://flexget.com/wiki/ChangeLog

Configuration

In order to use FlexGet you'll need to create a configuration file.

FlexGet uses YAML markup in configuration file. Here are some common mistakes:-

  • Indentation level. Always use (multiples of) 2 spaces and never use tab-key!
  • Plugins are supposed to be indented at the same level (rss, series, download etc). Some plugins may allow other plugins inside of them.
  • Missing colons. Pay special attention to these when looking at the examples and documentation.
  • If text value contains any of {}[]%: characters it must be quoted with .
  • If you want to pass a number as a text (ie. series 24), value must be quoted with .

Here is our example configuration:

~/.flexget/config.yml

feeds:
  tv-shows:
    rss: http://example.com/rss.xml
    series:
      - pioneer one
      - south park
    download: ~/torrents/

Now let's break that down line by line.

In first line feeds is a container which may contain any number of feeds under it. In our example there is only one feed called tv-shows. The feed name must be indented by 2 spaces because it belongs to feeds. In Yaml relations are represented by indentation level. Pay special attention to : - characters.

Fully Working Configuration Example

presets:
  tv:
    series:
      settings:
        720p:
          quality: 720p
        hdtv:
          quality: hdtv
      720p:
        - The Looney Tunes Show
        - Awake
      hdtv:
        # - Camelot:
            # watched: S01E03
        - The Devil You Know
        - Leverage
        - Wilfred
    email:
      from: name@myemail.com
      to:
        - me@myemailaddress.com
        - me@anotheremailaddress.com
    download: ~/Downloads/
feeds:
  ezrss-all:
    rss: http://www.ezrss.it/feed/
    preset: tv
  showrss-all:
    rss: http://showrss.karmorra.info/feeds/all.rss
    preset: tv
  # ezrss-camelot:
    # rss: http://www.ezrss.it/search/index.php?show_name=Camelot&mode=rss
    # preset: tv

Documentation

flexget --help
flexget --doc series

Thanks to the Flexget wiki

INFO: PLUGINS:

Our feed tv-shows has three plugins: rss, series, download. Notice all plugins again indented by additional 2 spaces. This is because they belong to the feed tv-shows. In correct configuration file all plugin keywords appear in the same level.

Common misconceptions

  • Plugin order doesn't matter, you can list them in any order you like. Most logical order would be inputs -> filters -> outputs.
  • Feed order doesn't matter, feeds are executed in seemingly random order. Use priority plugin prioritize feeds when necessary.
  • You cannot list plugins, i.e. rss twice in a single feed. This is limited by the chosen simpler syntax in the configuration file. Instead see preset plugin.

Sceper / Scene Releases

RSS

This is our input plugin which reads a RSS feed and produces processable entries. By looking at plugin rss documentation we learn that it accepts URL as a parameter in its simplest form.

Series

This plugin expects a list of series names as a parameter. Notice again how the list is intended by 2 spaces more than series keyword. The list belongs to series plugin. This filters entries, it goes through all of them and accepts those which match our interest.

Download

The process ends at output, in this case we have single output that just downloads all accepted entries into given path.

UPDATE - does not appear to work. Change to EZTV (see below).

Syntax:-

scenereleases: http://sceper.eu/category/tv-shows/tv-shows-x264?themedemo=SceneRLSv3

Full working configuration file example:-

feeds:
  tv-shows:
    scenereleases: http://sceper.eu/category/tv-shows/tv-shows-x264?themedemo=SceneRLSv3
    series:
      - The.Looney.Tunes.Show.2011.*720p
    download: ~/Downloads/

RSS + EZTV

feeds:
  tv-shows:
    # this is the actual feed you want, uncomment when ready
    # rss: http://www.ezrss.it/feed/
    # this is a test feed to see if it works, delete this when ready
    rss: http://www.ezrss.it/search/index.php?show_name=Camelot&mode=rss
    series:
      - Camelot
          # only anything newer than this will be downloaded
          watched: S01E03
    quality:
      min: hdtv
      max: 720p
    download: ~/Downloads/

Transmission RPC

Instead of downloading the .torrent file to a directory for Transmission to 'watch', this plugin passes the .torrent file to the Transmission daemon directly via RPC. Also, it supports the newer magnet urls.

First, install the Python Transmission RPC software:

sudo easy_install transmissionrpc

or

sudo pip install transmissionrpc

Then, update your FlexGet configuration file to include the new settings, such as transmission and set: path (which needs to be a directory writeable by transmission-daemon) as shown in the example below. Notice that you need to comment out the download option now which has been replaced by the set: path option.

presets:
  global:
    # download: ~/Downloads/
    transmission:
      host: localhost
      port: 9091
      username: username
      password: password
  tv:
    series:
      settings:
        shows:
          timeframe: 6 hours
          quality: 720p
          set:
            path: ~/Videos/transmission/seeding/

Now if you run a check you should see something like the following.

flexget --check
2012-02-17 23:40 INFO     transmission  feed-name      Trying to connect to transmission...
2012-02-17 23:40 INFO     transmission  feed-name      Successfully connected to transmission.

HOWTO: FIX:

ERROR: pip._vendor.requests.packages.urllib3.connection.VerifiedHTTPSConnection InsecurePlatformWarning

sudo -i
pip install urllib3[secure]
pip install --upgrade flexget

https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

https://urllib3.readthedocs.io/en/latest/user-guide.html#ssl-py2

ERROR: Series Behind

Show...

flexget series list

Error...

"! Latest download is 3 episodes behind, this may require manual intervention"

Fix...

flexget series begin "Kung Fu Panda: Legends of Awesomeness" S03E17

ERROR: pkg_resources.VersionConflict: (six 1.5.2

pkg_resources.VersionConflict: (six 1.5.2 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('six>=1.7.0'))

Fix...

sudo easy_install six==1.8.0

ERROR: BUG: Unhandled error in plugin transmission

2014-02-13 19:26 ERROR    task  kat-tv  BUG: Unhandled error in plugin transmission: 'Client' object has no attribute 'add_torrent'
Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/FlexGet-1.2-py2.6.egg/flexget/task.py", line 409, in __run_plugin
   return method(*args, **kwargs)

Fix...

easy_install --upgrade Flexget
easy_install --upgrade transmissionrpc

ERROR: No module named pkg_resources

sudo -i
curl -O http://python-distribute.org/distribute_setup.py
python distribute_setup.py
easy_install itsdangerous
easy_install markupsafe

ERROR: raise DistributionNotFound(req)

easy_install requests==0.10.0
sudo easy_install SQLAlchemy==0.7.4

http://flexget.com/ticket/1488

ERROR: pkg_resources.DistributionNotFound: python-dateutil<2.0

sudo easy_install python-dateutil==1.5

ERROR: pkg_resources.DistributionNotFound: FeedParser>=5.1.2

sudo easy_install FeedParser==5.1.2

ERROR: pkg_resources.DistributionNotFound: argparse

sudo easy_install argparse

Rejection

If FlexGet rejects your rules like this

tv-shows     Accepted: 0 (Rejected: 15 Undecided: 0 Failed: 0)

and you want to tell it to try them again, then run...

flexget --series-forget "Camelot"
Removed series `Camelot` from database.

Extracting Multi-Volume RAR Files

http://wiki.indie-it.com/index.php?title=Rar

RARFILE=$( ls *.rar ); echo "$RARFILE"; MKVFILE=$( rar l "$RARFILE" |grep 'mkv' |gawk '{ print $1 }' ); echo "$MKVFILE"; rar e "$RARFILE"; sleep 1s; mv "$MKVFILE" ~/Videos/

HOWTO: List Series In Config File

nano ~/.flexget/config.yml

    series:
   # START
     - APB  
     - Archer (2009)
     - Arrow  
     - Berlin Station
     - Big Little Lies
   # END
cat ~/.flexget/config.yml | sed -n '/\# START/,/\# END/p' | grep '-' | sed 's/^ *- //g'