Skip to topic | Skip to bottom
TWiki.BlackListPluginr1.17 - 29 Sep 2014 - 19:37 - Main.bscotttopic end

Start of topic | Skip to actions

Black List Plugin


This is a simple utility to keep black sheep away from a public TWiki site. The site can be protected against excessive page access (e.g. by bad robots), suspicious activities of users, such as multiple registrations or rapid topic updates indicating Wiki:WikiSpam, or saving text with wiki-spam.

The Plugin monitors activities by IP address and uses three IP address lists to protect the TWiki site:

  • WHITELIST: Manually maintained list of users who should never get on the BANLIST
  • BLACKLIST: Manually maintained list of malicious users
  • BANLIST: Automatically updated list of users with suspicious activities

On topic save, text is compared to a known list of spam patterns. If wiki-spam is identified, topic save is cancelled, an error message is shown, and the IP address is put on the BANLIST. Two wiki-spam lists are used:

  • Local SPAMLIST: Manually maintained list of spam patterns
  • Public wiki-spam list: Big list of wiki-spam patterns, retrieved from external web site

Users on the BLACKLIST and BANLIST will have every page access delayed by one minute and will get an error message.

The registration form can also be protected from improper use.

To fight Wiki-spam, the Plugin can also add a rel="nofollow" parameter to external URLs. Search engines will not follow links that have this parameter, taking away the incentive to add spam to TWiki.

Plugin Settings

Plugin settings are stored as preferences variables. To reference a plugin setting write %<plugin>_<setting>%, i.e. %INTERWIKIPLUGIN_SHORTDESCRIPTION%

General settings

  • One line description, is shown in the TextFormattingRules topic:
    • Set SHORTDESCRIPTION = Utility to keep malicious users away from a public TWiki site

  • Debug plugin: (See output in data/debug.txt)
    • Set DEBUG = 0

  • Log access of blacklist and spam list activities: (0 or 1)
    • Set LOGACCESS = 1


  • WHITELIST: Comma delimited list of IP addresses; possible to use partial addresses ending in a dot
    • Set WHITELIST =,,,
      • Here, we can document what/why (note that this is for people; TWiki only looks at the Set command)
      • localhost, default
      • home IP address of BruceDawson as of 1 Mar 2006
      • dynamic home IP address of BenScott as of 29 Sep 2014
      • static work IP for BenScott, 29 Sep 2014

  • BLACKLIST: Comma delimited list of IP addresses; possible to use partial addresses ending in a dot
    • Set BLACKLIST = 203.88.152., 203.88.155., 219.65.75.
      • Here, we can document what/why (note that this is for people; TWiki only looks at the Set command)
      • 203.88.152.: unknown, default
      • 203.88.155.: unknown, default
      • 219.65.75.: unknown, default

  • BANLIST: Automatically updated list of IP addresses based on BANLIST configuration
    • Action: IP address(es)
    • Current list:,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

  • BANLIST configuration, comma delimited list of:
    1. Points for registration
    2. Points for each save and upload
    3. Points for view and other actions
    4. Points for view raw
    5. Threshold to add to BANLIST
    6. Measured over time (in seconds)
    • Set BANLISTCONFIG = 20, 5, 1, 20, 120, 300
    • Your current score: 6 for IP address

  • Message for users on BLACKLIST and BANLIST:
    • Set BLACKLISTMESSAGE = Your IP address is black listed at the GNHLUG web site due to excessive access or suspicious activities. Please contact site administrator if you got on the list by mistake.

Wiki-spam filtering settings

  • Filter wiki-spam on topic save and HTML file uploads based on SPAMLIST: (0 or 1)

  • Comma separated list of Web.Topics to exclude from wiki-spam filtering on topic save:

  • Local SPAMLIST: Manually maintained list of spam patterns (NOTE: Must be RegularExpression patterns)
    • Action: wiki-spam regex pattern(s) http://.*?
    • Current list:

  • Cache refresh time (in minutes) for public wiki-merge pattern list:

  • Cache refresh time (in minutes) for internal wiki-spam regular expression cache:

  • Message for users trying to save text with wiki-spam:
    • Set WIKISPAMMESSAGE = Wiki-spam detected: "%WIKISPAMWORD%" is a banned word and cannot be saved. Your IP address is black listed at the GNHLUG web site due to suspicious activities. Please contact site administrator if you got on the list by mistake.

Registration protection settings

  • Protect registration: (number of minutes to expire, 15 minutes is recommended, 0 to disable)
    • Set REGEXPIRE = 15
    • If enabled, a magic number is protecting the registration process. TWiki expects a form field with a magic number. An error message is shown if not valid or if expired. This makes it harder to register a user by a script. A hidden field needs to be added to the registration form:
      <input type="hidden" name="rx" value="%BLACKLISTPLUGIN{ action="magic" }%" />

  • Message shown when using registration form incorrectly: (this message is deliberately vague)
    • Set REGMESSAGE = Registration failed, please try again.

Nofollow link setting

  • Add a rel="nofollow" parameter to external URLs. Use this is to fight Wiki-spam. Search engines will not follow the link if a URL has a nofollow parameter, such as <a href="" rel="nofollow">. Specify topic age in hours for which the nofollow parameter should appear (set it to a value that gives you enough time to remove spam); set it to -1 to add the nofollow parameter unconditionally to external URLs; or 0 to disable: (-1, 0, 1...N)
    • Set NOFOLLOWAGE = 168

Plugin Installation Instructions

Note: You do not need to install anything on the browser to use this plugin. The following instructions are for the administrator who installs the plugin on the server where TWiki is running.

  • Download the ZIP file from the Plugin web (see below)
  • Unzip in your twiki installation directory. Content:
    File: Description:
    data/TWiki/BlackListPlugin.txt Plugin topic
    data/TWiki/BlackListPlugin.txt,v Plugin topic repository
    lib/TWiki/Plugins/ Plugin Perl module
    pub/TWiki/BlackListPlugin/.htaccess Apache access control to protect pub dir
    templates/oopsblacklist.tmpl Generic oops message
  • Write protect this Plugin topic (if not already done)
  • Make sure pub/TWiki/BlackListPlugin/ is writable by the CGI user (typically nobody)
  • Add the hidden form field to the registration form as described in the "Registration protection settings" section (this is already in TWiki 4.0)
  • From TWiki 4.0 release on: Run the configure utility in your browser to enable the Plugin
  • Test if the installation was successful:
    1. Using above form, add the IP address of one of your workstations to the BANLIST
    2. Access TWiki from that workstation
      • if you look at a TWiki topic (with the view script) you should see the BLACKLISTMESSAGE (defined above) after a one minute timeout
      • else, you should get an 500 Internal Server Error for other scripts
    3. On a different workstation, remove the IP address of the test workstation from the BANLIST

Known Issues and Limitations

  • Wiki-spam filtering for HTML attachments works in TWiki 01-Sep-2004 and later.
  • With TWiki 4.0.2 on some platforms, notably Solaris, attached files are uploaded with a zero file size. This is because there is a bug in how TWiki 4.0.2 handles the beforeAttachmentSaveHandler. If affected, upgrade TWiki or apply bug fix Item2390

Plugin Info

Plugin Author: TWiki:Main/PeterThoeny
Plugin Version: 02 Jun 2006 (r10442)
Change History:  
02 Jun 2006: Added wiki-spam filtering for HTML attachments to combat TWiki:Codev.HtmlAttachmentSpam
29 Apr 2006: Added %BLACKLISTPLUGIN{ action="spam_show_n" }% that shows the local spam list in a shareable format with newline separator
07 Feb 2006: TWiki Release 4.0 fix to allow registration with e-mail verification, reset password and approve
03 Jan 2006: Filter lines with space from spam list; fixed bug that inproperly filtered HTML from spam list; Dakar Release fix (end/postRenderingHandler issue)
08 Nov 2005: Doc fixes; code warning fixes; allow empty local SPAMLIST and public spam list
04 Nov 2005: Added registration protection with magic number
30 Oct 2005: Dakar Release compatibility: Work around Dakar preferencs bug
29 Oct 2005: Added wiki-spam filtering to prevent topic save with wiki-spam
27 Oct 2005: For BANLIST, add/remove multiple IP addresses at once, contributed by TWiki:Main.MichaelDaum
22 Jan 2005: Added NOFOLLOWAGE handling
19 Jan 2005: Added score for "view raw" to address e-mail harvester issue
05 Apr 2004: Fixed bug in event log (requiring update of earlier Plugin versions); doc updates
04 Apr 2004: Added WHITELIST and BANLIST
21 Mar 2004: Initial version
CPAN Dependencies: none
Other Dependencies: none
Perl Version: 5.005
License: GPL (GNU General Public License)
TWiki:Plugins/Benchmark: GoodStyle 99%, FormattedSearch 99%, BlackListPlugin 97%
Plugin Home:

Related Topics: TWikiPreferences, TWikiPlugins, Google wiki spam directory

-- TWiki:Main/PeterThoeny - 02 Jun 2006
to top

You are here: TWiki > BlackListPlugin

to top

All content is Copyright © 1999-2023 by, and the property of, the contributing authors.
Questions, comments, or concerns? Contact GNHLUG
Legal Notice (includes Terms of Service)