Introducing: K2

SVN: http://svn.xp-dev.com/svn/k2/
Readme: http://svn.xp-dev.com/svn/k2/README <DO IT

This is essentially a crawler written nearly from scratch (save for reusing some usual libraries until I can find better). It works in a very crude fashion in comparison to Ibitzy, but I think it gets the job done. It can easily queue up 300 torrents in a minute, processing them (get the torrent, decode some stuff, scrape, insert) will obviously take longer, but it uses cURL and MySQLi to help with _some_ performance.

Enjoy.

Edit



ViewVC: http://svn.kosmodrom.64x.org/viewvc/k2/trunk/
Tarballed trunk: http://svn.kosmodrom.64x.org/viewvc/k2/trunk/?view=tar

This post has been edited by Geeker: Oct 28 2009, 09:58 AM

Nice. But you should have made it a plugin based system like Ibitzy.

This post has been edited by major9: Oct 27 2009, 05:15 PM

Agreed, even though this is only a small preview, I do plan on adding some functionality between or for more sites (IsoHunt being my biggest goal) at some point here soon. The other thing I'm debating is an actual front-end, but I don't want this to be another all-in-one indexer, so that probably won't happen.

Also, (if a mod wouldn't mind editing the OP, since I can't anymore), I've moved the repository over to my site:


ViewVC: http://svn.kosmodrom.64x.org/viewvc/k2/trunk/
Tarballed trunk: http://svn.kosmodrom.64x.org/viewvc/k2/trunk/?view=tar

[Edit]

Revision 5 contains working mininova and Isohunt plugins.

This post has been edited by phrostwave: Oct 28 2009, 08:40 AM

Can i use this whit xbtit or is there an version of ibitzy avabile.

You'll need to make some changes for XBTIT. You can ask me to do the custom work

You could easily modify this to work with xbtit, although I don't see the point in doing so since you're already running a tracker.

Also, revision 10 now contains a BTJunkie plugin. This is interesting because there are now two methods to retrieve data from an XML feed:

"Atom" style: Isohunt and Mininova use this, using an "enclosure" node tag with an attribute containing the torrent url
"RSS" style: BTJunkie uses this, using the standard "item" parent node tag with a child tag ("guid", in this case) containing the url (which needs to be transformed slightly to point to the correct location).

TL;DR: Basically, you can read and import from two different styles of XML feeds.

[Edit]

Revision 15 seems to hold its own, so I've branched this off into version 0.1:

ViewVC: http://svn.kosmodrom.64x.org/viewvc/k2/branches/0.1/
Tarball: http://svn.kosmodrom.64x.org/viewvc/k2/bra...s/0.1/?view=tar

This post has been edited by phrostwave: Oct 29 2009, 06:30 AM

Some sites have a firewall kind of thing blocking if you download more than a certain no. of torrents in a certain time.

Ironic, because some of their HTML pages they serve are larger than the torrent files themselves. Also, BTJunkie and bittorrent.am plugins will ban you from crawling new torrents, but hey, those two sites index others and bump their success off others bandwidth, so I'll make sure to return the favor right back at 'em

Plugins looks much more easier than Ibitzy.

Some recent changes up to Revision 31 & Branch 0.3:

- includes 7 example plugins
- no longer depends on the two class files
- inserts announce-list in serialized format (use unserialize($data) to flop it back into a normal array)
- inserts the contained metadata total filesize, serialized file paths / names / lengths, torrent creation date, and what it was created by.

The announce-list is yet to be used for mutli-tracker scraping on a single torrent, but I plan on making it use the first tracker that responds.

[Edit]
Revision 39 of the trunk can pretty much support multiple trackers, but note that it breaks when the first tracker responds at this revision.
Also, Rev 39 was branched into 0.4: http://svn.kosmodrom.64x.org/viewvc/k2/branches/0.4/

And for the record, I've been able to queue up roughly about 400 torrents in under 30 seconds, and process around 100 in 3-5 minutes, depending on everyones overall traffic.

This post has been edited by phrostwave: Nov 2 2009, 03:01 AM

I noticed some important parts of the plugin in k2-queue.php. I think you should move those to the corresponding plugin files. It'll help the people who develop additional plugins if all plugin codes are centralized.

Revision 72 seems to be pretty stable (save for really strange scrape responses, and indexers that ban), a new feature being that it is now capable of caching whatever you have crawled at torrage (info at http://torrage.com/automation.php), and the export links back to the file hosted on torrage.

Also, I might be adding the above plugin layout at some point.

This post has been edited by phrostwave: Nov 7 2009, 06:49 AM

Where can i download the lastest version of K2?
The site (http://svn.kosmodrom.64x.org/viewvc/k2/branches/0.4/) is down/offline.

Whats the main difference between K2 and Ibitzy?

svn down

@BTLoader - Ibitzy hasn't been updated for a while so none of the plugins work.

I know that the current version of Ibitzy is old, it was not what I asked.
I asked what the difference was between Ibitzy and K2.
And the mininova-plugin works, but not the categories.

K2 has more features, I think.
Btw. Mininova plugin is useless now as they've removed all their torrents except for their Content Distribution ones.

This sounded interesting and was going to check it out but -

Firefox can't find the server at svn.kosmodrom.64x.org. or

Authorization Required

This server could not verify that you are authorized to access the document requested. Either you supplied the wrong credentials (e.g., bad password), or your browser doesn't understand how to supply the credentials required.
Apache/2.2.11 (Ubuntu) DAV/2 PHP/5.2.6-3ubuntu4.4 with Suhosin-Patch mod_ssl/2.2.11 OpenSSL/0.9.8g mod_wsgi/2.3 Python/2.6.2 SVN/1.6.6 Server at svn.xp-dev.com Port 80


if you don't wish someone to check it out why post it ?
regards
thebrass

Any one got a copy of this pls, could really do with it..

thx

This post has been edited by nat98: Feb 16 2010, 11:42 PM