WebSeitz/wiki

WikiGraph

MetaWiki-like service delivered via JavaScript. See 2012-12-01-BuildingWikiGraph

Short-term goals: Visible Backlinks and Twin Pages for this WikiLog.

Longer-term goals

Building with WebPy to run on my Linode.

Data in 3 SQL tables: spaces, pages, mentions

Scraped page lists by hand from a number of my Sister Sites

For Visible Backlinks, started with 2003 file giving graph of my space (only 2804 pages at that time!) - good enough for starting to build.

Jan20'2013 - get static pieces working together

MoinMoin

  • edit /MoinMoin/theme/__init__.page_info()
            html += """\n<div id="backlinks"></div>
				<script src="http://localhost:8081/backlinks.js"></script>
				<noscript>Please enable JavaScript to view the BackLinks</noscript>"""

            html += """\n<div id="twin_pages"></div>
				<script src="http://localhost:8081/twinpages.js"></script>
				<noscript>Please enable JavaScript to view the Twin Pages</noscript>"""

WebPy

  • routes
urls = ( 
	'/twinpages.js','twinpages',
	'/backlinks.js','backlinks',
  • code
class backlinks:
    def GET(self):
        web.debug('calling backlinks')
        resp = ''
        backlinks = db.select('mentions', where="space_name = 'Web Seitz Wiki' AND page_mentioned = 'Bill Seitz'")
        web.debug('backlinks: %s' % (backlinks))
        if bool(backlinks):
            resp = 'BackLinks: '
            for backlink in backlinks:
                resp += '<a href="%s%s">%s</a> | ' % ('http://webseitz.fluxent.com/wiki/', backlink.page_name, backlink.page_name)
            resp = resp[:-3]
        else:
            resp = 'No backlinks!'
        web.debug(len(resp))
        return """
        document.write('\
        <div>\
        <p>%s</p>\
        </div>');
        """ % (resp)

Feb22: clarify MVP plan

Feb26: buy ThirdPartyJavaScript book by Ben Vinegar and Anton Kovalyov (DisQus guys)

Feb27: get dynamic version of code working, using Http Referer and document.write()

Mar01: done styling content with inline div styles (how tacky).

Next: figure out how to parse all my current WikiLog pages for Visible Backlinks database.

  • Mar10: done writing/testing locally
  • Mar11: running against production.
    • a few weird (MoinMoin) cases where current filename contents point at a revisions/ file that doesn't exist! Always a tuggle in these cases as to how hard to work at solving the general problem vs just hacking the exceptions and re-running.
    • have forward-refs for 15801 pages.

Mar12: suck 96634 mentions into local db

  • have some weird UTF8 issues, cleaned up file by hand

Next:

Setting up hosting

Make a /srv/www/wikigraph.net/* beside /srv/www/simplest-thing.com/* - do chown/chmod bits.

  • cp /srv/.../app/start_uwsgi_app.sh to parallel directory. Edit (port 9002).

Copy /etc/default/uwsgi to /etc/default/uwsgi_wg, edit the latter to set PYTHONPATH to /srv/www....

Edit /opt/nginx/conf/nginx.conf - add a header section for hostless-domain redirect and a header for the www.wikigraph.net host that points to the right place and uses port 9002 for uwsgi.

Copy WebPy code.py to server.

Restart nginx. Kill/restart the uwsgi app for Simplest Thing. Start the uwsgi_wg app for WikiGraph.

Try to hit http://www.wikigraph.net/test - get "uWSGI Error - Python application not found"

  • hit same point in setting up Simplest Thing, but had already handled tweaks from that learning here, so don't know what issue is...

Tweak start_uwsgi_app.sh to call uwsgi_wg instead of just uwsgi. Kill/restart uwsgi.

Try to hit http://www.wikigraph.net/test - get 502 Bad Gateway. Realize didn't get the new uwsgi processes. Change the bash back. Get the "uWSGI Error" again.

Check the logs

  • access.log - looks like status 500
  • uwsgi_app.log - ah, notes that import forms failed (because not using that in this app yet, so hadn't copied it to server).

Tweak code.py, copy up, kill/restart uwsgi, hit page again. It does a redirect to /static/... based on code copied from Simplest Thing, but I don't have that directory, so get 404 error.

Comment-out that check_authorized code, copy/restart/etc - now works!

Mar14: convert SQLite dump file to PostgreSQL

Mar20

Apr15: tweak to let my local Private Wiki call it. Automatically works for creating-new page process so I can see if Twin Page already exists in my WikiLog and decide whether to add a Private page or stay public.

Future:

Jul'2013 update: tweaked scraper code to catch some cases being missed before, re-ran

  • 15963 pages
  • 100451 mentions (6.3 mentions/page) (was 6.1 before - of course since both pieces changed it's hard to be sure...)

Twin Pages: pondering addition of Wikipedia

  • 5M English articles, so don't feel like doing that
  • 1000 "vital" articles, but suspect most wiki sites wouldn't have matches against very many of those (but maybe that's ok)
  • could also do a hack, and always include a Wikipedia link that just generates a search...
  • since Wikipedia is so unlike every other wiki, it actually sounds like a bad fit

Edited:    |       |    Search Twitter for discussion

Made with flux.garden